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OVERVIEW 



THE SENIOR URBAN EDUCATION 
RESEARCH FELLOWSHIP PROGRAM 

Large urban public school districts play a significant 
role in the American education system. In 2007-08, 
the largest 65 urban school systems in the country 
- comprising less than one half of one percent of the 
nearly seventeen thousand school districts that exist 
across the United States - educated about 1 4 percent of 
the nation's K-12 public school students, approximately 
one third of its African American students, a third of 
its English Language Learners, a quarter of its poor 
students, and a quarter of its Hispanic students.' They 
also employed nearly 1 5 percent of the nation's teaching 
force. Clearly, any attempt to improve achievement and 
to reduce racial and economic achievement gaps across 
the United States must involve these school districts as a 
major focus of action. 

These school districts face a number of serious, systematic 
challenges. However, solutions to these problems are 
not always obvious, and the existing research base is 
not always sufficient to address them. In order to better 
understand the problems in urban education and to 
develop more effective and sustainable solutions, urban 
districts need a program of rigorous scientific inquiry 
focusing on what works to improve academic outcomes 
in the urban context. Moreover, in order to produce such 
evidence and to move public education forward generally, 
the standards of evidence in education research must be 
raised in such a way as to bring questions regarding the 
effectiveness of educational interventions and strategies 
to the fore and to promote careful scrutiny and rigorous 
analysis of the causal inferences surrounding attempts 
to answer them. 

It has been argued that, in order to move such an effort 
forward, a community of researchers, committed to a 
set of principles regarding evidentiary standards, must 
be developed and nurtured. We contend further that, in 
order to produce a base of scientific knowledge that is 
both rigorously derived and directly relevant to improving 
achievement in urban school districts, this community of 
inquiry must be expanded to include both scholars and 
practitioners in urban education. 



Though a great deal of education research is produced 
every year, there is a genuine dearth of knowledge 
regarding how to address some of the fundamental 
challenges urban school districts face in educating 
children, working to close achievement gaps, and 
striving to meet the challenges of No Child Left Behind. 
Moreover, while there is a history of “process-related” 
research around issues affecting urban schools, relatively 
few studies carefully identify key program components, 
document implementation efforts, and carefully examine 
the effects of well designed interventions in important 
programmatic areas on key student outcomes such as 
academic achievement. In sum, there is an absence of 
methodologically sound, policy relevant research to help 
guide practice by identifying the conditions, resources, 
and necessary steps for effectively mounting initiatives 
to raise student achievement. 

In order to address this need, the Council of the Great City 
Schools, through a grant from the Institute of Education 
Sciences, established the Senior Urban Education 
Research Eellowship (SUERE) program. 

The Senior Urban Education Research Eellowship was 
designed to facilitate partnerships between scholars and 
practitioners focused on producing research that is both 
rigorous in nature and relevant to the specific challenges 
facing large urban school districts. We believe such 
partnerships have the potential to produce better, more 
practically useful research in at least three ways. First, 
by deepening researchers’ understanding of the contexts 
within which they are working, the program may help them 
maximize the impact of their work in the places where it is 
needed the most. Second, by helping senior staff in urban 
districts become better consumers of research, we hope 
to increase the extent to which the available evidence 
is used to inform policy and practice, and the extent 
to which urban districts continue to invest in research. 
Third, by executing well designed studies aimed at the 
key challenges identified by the districts themselves, we 
hope to produce reliable evidence and practical guidance 
that can help improve student achievement. 



1 Council of the Great City Schools (2010). Beating the Odds: An Analysis of Student Performance on State Assessment and NAEP. Results from the 2008-09 School Year. Washington, DC. 



The Council of the Great City Schools 






The primary goals for the Senior Urban Education 
Research Fellowship are to: 

• promote high quality scientific inquiry into the ques- 
tions and challenges facing urban school disfricts; 

• facilitate and encourage collaboration, communi- 
cation, and ongoing partnerships between senior 
researchers and leaders in urban school districts; 

• demonstrate how collaboration between scholars 
and urban districts can generate reliable results 
and enrich both research and practice; 

• produce a set of high quality studies that yield 
practical guidance for urban school districts; 

• contribute to an ongoing discussion regarding 
research priorities in urban education; and 

• promote the development of a "communify of 
inquiry”, including researchers and pracfitioners 
alike, committed to both a set of norms and prin- 
ciples regarding standards of evidence and a set 
of priorities for relevanf, applied research in urban 
education. 

The following volume of the Senior Urban Education 
Research Fellowship Series documents the work of 
Dr. Robert Meyer, Dr. Bradley Carl, and Huiping Emily 
Cheng, working in collaboration with the Milwaukee 
Public Schools (MPS). Both the research and reporting 
is the sole intellectual property of Dr. Meyer, and reflects 
his personal experience and perspective as an education 
researcher working in collaboration with MPS. 

Dr. Meyer and his team have produced strong new work 
in the area of “early warning systems” for identifying 
students at risk for adverse high school outcomes. We are 
particularly pleased to see how the relationship between 
the Value Added Research Center at the University of 
Wisconsin-Madison has grown, and is continuing to grow. 
Their experience working hand in hand with the district, 
and the district’s consistently active and open support of 
this work, offers us one of the best models for disfrict- 
researcher collaborafions in fhe country. 



Their extensive understanding of the patterns of 
academic failure within the district— as well as the data 
needs of MPS to address these challenges— have led 
to some particularly exciting innovations in this area. 
For example, they have built on the “on-track indicator" 
work of the Consortium on Chicago School Research 
to create a new measure— the “Total Quality Credits” 
indicator, a continuous variable of student achievement 
that measures not only whether a student is “on track” 
to graduate or not, but how far on or off frack they 
are by the end of their first year of high school. This 
measure provides the district with crucial information for 
understanding a student’s educational progress in more 
detail. 

The research team also takes a step beyond high school 
graduation to illustrate that post-secondary outcomes— 
the skills with which students graduate and rates of 
enrollment in college— are as relevant and important a 
target for district efforts as high school graduation. While 
they acknowledge that the currently available measures 
of post-secondary outcomes are insufficient in MPS— i.e., 
enrollment in college rather than college persistence and 
graduation — this push for farther-reaching measures of 
how well our public schools are serving sfudenfs is both 
timely and important. We hope you will find it interesting 
and relevant to your own work. 

Thank you. 

Michael Casserly 

Executive Director 

Council of the Great City Schools 
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ABOUT THE RESEARCH PARTNERSHIP 



THE HISTORY AND EVOLUTION OF THE MPS/ 
WCER RESEARCH COLLABORATION 

The Senior Urban Education Research Fellowship has 
provided a key opportunity to build upon an already- 
strong research partnership between the Milwaukee 
Public Schools (MPS) and the Wisconsin Center for 
Education Research (WCER). As a part of UW-Madison's 
School of Education, WCER provides a productive 
environment where scholars conduct basic and applied 
education research. WCER research spans the full 
scope of education, from the effects of infanf child 
care and after-school programs to undergraduate and 
graduate curriculum reform. Housed wifhin WCER, fhe 
Value-Added Research Center (VARC) is dedicated to 
developing, applying, and disseminating value-added 
and longitudinal research methods to evaluate the 
performance and effectiveness of schools and teachers, 
as well as educational programs and policies. Under fhe 
leadership of VARC director Robert H. Meyer, a staff of 
faculty, scientists, researchers, education professionals, 
and data system specialists provide a wide variety of 
cross-disciplinary expertise to allow for groundbreaking 
work on value-added systems, program and policy 
evaluation, data-driven decision making, and education 
reform. 

With a current enrollment of some 82,000 students - 
making it by far Wisconsin's largest district and among 
the largest in the country - MPS is a natural research 
partner for WCER’s work. The MPS/WCER research 
partnership dates back to collaborative work on the 
development of district-level standards and assessments 
in the mid-1990s, and expanded significantly with 
VARC's production of the first set of MPS value-added 
reports at the school and grade level for the 2001-02 
school year. These reports were based upon data from 
fhe Wisconsin Knowledge and Concepts Examination 
(WKCE) and the TerraNova assessment series. MPS, 
along with Minneapolis and Dallas, was among VARC's 
first partner districts for value-added work; this work has 
since expanded to include two of the largest districts in 
the country (the New York City Department of Education 



and the Chicago Public Schools), in addition to nearly 40 
Wisconsin districts of varying sizes above and beyond 
MPS (including fhe Madison Metropolitan and Racine 
Unified school districts) which collectively enroll more 
than 30% of the total K-12 enrollment in Wisconsin. 
VARC also has an ongoing contract to provide technical 
assistance to districts across the country that are 
implementing value-added as one component of teacher 
merit pay plans under the federally-funded Teacher 
Incentive Fund (TIE) program.^ 

As of this writing in the spring of 201 0, VARC has recently 
delivered its 9th annual set of value-added reports for 
MPS based upon WKCE data through the 2009-10 
school year. The district has continued its practice of 
posting all value-added reports publicly on the Web site 
of the Division of Research and Assessment, and has 
incorporated value-added data into a variety of district and 
school improvement initiatives. Value-added data were a 
major component, for example, in the selection of a group 
of higher-performing schools (known as Mosaic schools) 
which received increased autonomy in matters related to 
curriculum and budgets, and were also used to identify 
a second set of lower-performing schools that received 
additional targeted assistance from fhe district as well 
as the National Education Association Foundation. Value- 
added data are also a required component of the School 
Improvement Plans submitted by each school on an 
annual basis to identify areas of strength and weakness 
in terms of planning for future improvement. 

With VARC's rapidly-increasing profile in terms of 
producing value-added information has come the 
realization that value-added data must be accompanied by 
interpretive information and professional development 
in order to be maximally useful as a school improvement 
tool. To this end, VARC has worked collaboratively with 
its research partners— including MPS— to create to 
create a series of professional development materials 
and activities which include both on-site presentations by 
VARC staff to build a “train-the-trainers” model as well as 
on-demand content accessible through the VARC Web 
page.^ 



2 See httpV'/www.cecr.ed.gov and http://varc.wceruw.org/Projects/center_educator_comp_reform.php. 

3 Several of the professional development materials produced for use in MPS have been made available to other districts in Wisconsin and across the country via the “Tutorials" section of the 
VARC Web site; see http://varc.wceruw.org/tutorials/index.php. 



The Senior Urban Education Research Fellowship Series, Volume II - Fall 2010 






ABOUT THE RESEARCH PARTNERSHIP 



VARC's value-added work with MRS and other partner 
districts has also recently expanded to include pilot 
versions of value-added data based upon short-cycle 
(benchmark or interim) assessments administered three 
or four times annually, as well as selected non-cognitive 
measures such as attendance. There is great interest in 
this type of new value-added data generated from short- 
cycle assessments, as these data can be used far more 
readily for school improvement purposes than can data 
from summative tests due to the “time lag” involved in 
summative tests (in Wisconsin, for example, results from 
the state accountability test taken in November each year 
are not made available to districts until March). Short- 
cycle assessments also alleviate the problem of needing 
to apportion growth across two separate academic 
years, which is made necessary by mid-year testing in 
Wisconsin and other states. 

Above and beyond foundational work with value-added 
indicators as described above, the MPS/WCER research 
partnership has expanded substantially in recent years 
to include a wide array of program evaluation and data- 
driven decision-making work, as well as efforts to build 
greater capacity in the areas of human capital (including 
district and school leadership and the district's employee 
performance evaluations) and performance management. 
In the area of program evaluation, MRS has awarded 
contracts to VARC to evaluate and/or design a number 
of major district initiatives in recent years, including: 

• Read 1 80 (an intervention for lower-performing 
readers in middle and high school) 

• Charter schools 

• Literacy Coaches 

• Supplemental Educational Services (SES) providers 

• Revised principal and teacher evaluation systems 

• Instructional walk-throughs 

• Uses of School Improvement Plans 

• Credit recovery and “Freshman Connection” pro- 
grams for high school students 



In the area of data usage and data-driven decision- 
making, MRS and VARC are collaborating on the 
development of an Integrated Resource Information 
System (IRIS), which has two related goals: to assess 
student, teacher, classroom, and school effects on value- 
added student learning gains; and to connect resources at 
the school, classroom, and student levels to effectiveness 
in improving student learning. 

VARC staff also worked actively with MRS during the 
2009-10 school year on a comprehensive reform of 
the district's employee evaluation system for principals 
and teachers, as well as the implementation of 
Performance Management principles for improvement 
and accountability. Specific initiatives include the 
implementation of the “EdStat” model for data review 
and continuous improvement. By focusing attention on 
a selected set of “Key Performance Indicators,” or KPIs, 
and providing a regular time for district and school staff 
to reflect on progress along these measures, the EdStat 
model provides a useful new way of incorporating data 
into the school improvement process in MRS. 

An additional, very unique aspect of the MPS/WCER 
relationship is the “embedded researcher” concept. 
Under this arrangement, which was initiated in 2005, a 
WCER employee splits time each week between WCER 
in Madison and the Division of Research and Assessment 
at MRS, working under the technical direction of VARC 
director Dr. Robert Meyer and under the functional 
direction of MRS director of Research and Assessment 
Deb Lindsey. This arrangement provides WCER with a 
constant presence in the state's largest school district, 
which is invaluable in continuing and strengthening 
the university-district relationship, and provides MRS 
with on-demand capacity in matters of research and 
program evaluation that is rare even in the largest urban 
districts. The current embedded researcher. Dr. Bradley 
Carl, assists MRS with ongoing initiatives such as the 
development and refinement of the vertical scale for 
the MRS benchmark assessment system as well as 
evaluations of MRS programs. 
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One of the ways in which the Senior Urban Fellowship 
has been particularly useful is that it has fit very nicely 
with, and contributed greatly to, a substantial expansion 
of the existing MPS/WCER research partnership to the 
high school level. Many of the projects noted above, 
from VARC's core value-added work to the specific 
evaluation projects undertaken by VARC staff in MPS, 
have appropriately focused on the elementary and middle 
school levels, as these have been areas of high interest 
to the district and have the advantage of having pre- 
existing data sets (often in the form of test scores) readily 
available for evaluation purposes. At the same time, 
however, there has been a greatly increased emphasis 
in the recent past on high school-related outcomes as a 
portion of the MPS/VARC research agenda, both from 
the district’s perspective as well as VARC’s. Specific 
examples here include the following: 

• A comprehensive evaluation of MPS high schools, 
focusing on differences across high school types 
(small vs. large, charter vs. non-charter, etc.) with 
respect to key performance indicators including 
test scores, attendance, and initial postsecond- 
ary enrollment; the final report was submitted in 
December 2009 and presented to the MPS Board 
of Directors in February 2010. 

• Tracking postsecondary outcomes, including initial 
college enrollment and rates of persistence among 
MPS graduates, for recent cohorts of MPS gradu- 
ates 

• Evaluating the impact of several new initiatives 
(funded by federal stimulus dollars) intended to 
improve outcomes for MPS freshmen, including a 
series of “freshman connection" activities designed 
to improve transitions from middle grades to high 
school as well as a new credit recovery project 
aimed at reducing the high rate of MPS students 
who must repeat grade 9 each year. This work, 
launched in late 2009, will produce an interim 
report in June 2010, followed by a second year of 
work that will end with a final project report a year 
later. 
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EXECUTIVE SUMMARY 



This report summarizes work conducted to date through 
the Senior Urban Education Research Fellowship 
(SUERF) awarded by the Council of the Great City 
Schools to the Value-Added Research Center (VARC) 
at the University of Wisconsin-Madison for work in the 
Milwaukee Public Schools (MPS). VARC has utilized 
its Fellowship award, entitled “Accountability and 
Performance in Secondary Education in the Milwaukee 
Public Schools,” to substantially expand an already-strong 
research partnership with MPS into exciting new work at 
the high school level. 

Specifically, the SUERF award has supported two key 
areas of inquiry related to high school performance 
and accountability in MPS. The first, which is the major 
focus of this paper, is the development of an “early 
warning” system to help MPS identify middle and high 
school students at elevated risk of two key adverse 
outcomes: (a) failing to graduate from high school, and 
(b) graduating, but with low levels of college/workforce 
readiness. The second, as summarized in the Appendix, 
is the development of a new class of value-added 
performance indicators for MPS high schools, including 
the use of non-cognitive outcomes such as attendance 
to complement more traditional (test-based) value-added 
metrics. 

METHODOLOGY 

Part 1. Overview of Data Used and Challenges 
Encountered 

A guiding principle in creating the MPS early warning 
system has been to gather data from multiple cohorts 
of first-time MPS 9th grade students and work both 
“backward” (e.g., into middle school) as well as “forward” 
(into high school) to develop predictive models for student 
success on the primary outcomes of interest (high school 
graduation and college enrollment). Assembling a full 
“panel” of data to describe the complete middle and 
high school experiences of MPS students presented a 
significant challenge, both because MPS data sets are 
somewhat limited prior to the 1998-99 school year 
and because this task requires nearly 1 0 years of data 
(including at least three years of middle school as well 
as 4-6 years of high school after accounting for student 



retentions). The primary solution to this problem was a 
recursive “data splicing” approach which relied upon 
complete sets of middle school outcomes for certain 
cohorts of students combined with different sets of 
complete high school outcomes for other cohorts. These 
disparate data sets were then linked together through the 
common metric of Total Quality Credit (TQC) attainment 
during the first year of high school, as described In more 
detail below. Data for the second major outcome being 
predicted (college enrollment) was provided by accessing 
National Student Clearinghouse (NSC) files, which MPS 
began acquiring during the 2008-09 school year. 

Part 2. Data Inventory: What is Known about 
MPS Dropouts? 

A key starting point for the design and features of the 
MPS early warning system was a series of descriptive 
findings related to MPS dropouts and graduates. This Is 
not because the descriptive findings were surprising per 
se, but rather because they provide important information 
regarding which factors yield the greatest predictive 
power for understanding which students are least likely 
to graduate and attend college, as well as how to weight 
these predictors relative to one another. The key findings 
are as follows; 

How early into their high school career do most MPS 
students tend to drop out? 

• A relatively small number of MPS students who 
eventually dropped out (7.1 % across the two 
cohorts combined) did so during their first year of 
high school. It Is more common for MPS students 
to drop out during their third and fourth years of 
high school, although a surprising number of MPS 
students make If to their fifth and even sixth year of 
high school before dropping out. As has been ob- 
served in other cities, it appears that dropping out 
of MPS as an outcome is best understood as a 
gradual process of disengagement and academic 
shortcoming that often plays out across multiple 
years of a student's high school career, rather 
than as a single catastrophic event that leads to 
immediate withdrawal from school. 
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• Despite the fact that many MRS students persist 
into their fourth, fifth, and even sixth'' year of high 
school before dropping out, the largest group of 
dropouts in terms of students’ last grade prior to 
dropping out was 9th graders, who were 26.0% 
of all dropoufs for the two cohorts combined. From 
a data standpoint, this confirms the need to treat 
grade in high school as a completely different vari- 
able than amount of time spent in high school, as 
many students are retained in 9th grade once, and 
even twice in a substantial number of cases, before 
dropping out. 

ffow do eventual dropouts and graduates differ with 
respect to key outcome indicators during the first year 
of high school and middle school years? 

• Students who eventually dropped out of high 
school had substantially lower academic perfor- 
mance (GPA) in core academic subjects (reading, 
math, science, and social studies) in their first year 
as 9th graders, and failed a much higher percent- 
age of core subject courses than did eventual 
graduates. 

• Dropouts were much more likely to have been 
retained in grade 9 than graduates. 

• The absence rates of eventual dropouts were much 
higher than those of eventual graduates, both on a 
year-end as well as a month-to-month basis. 

• Eventual dropouts as a group had higher inci- 
dences of behavioral problems than did eventual 
graduates, both in terms of how often they were 
suspended as well as the severity of the offenses 
committed. 

• Standardized test scores for eventual dropouts 
were lower than for eventual graduates, although 
test scores provide comparatively little predic- 
tive power, particularly toward graduation as an 
outcome. 



• Dropouts were more mobile than graduates, as 
measured by the frequency of month-to-month 
changes in enrollment during the school year, 
although mobility overall yielded comparatively 
little predictive power after other variables were 
controlled for. 

Part 3. Creating the “Total Quality Credits” 
Indicator 

Given the importance of the first year of high school, 
the utility of an “on track” measure to classify students 
according to credit accumulation and number of failing 
grades in core courses at the end of the first year was 
evident. However, there are two main limitations to such 
a measure. First, classifying students as either on-track 
or off-track at the end of the first year of high school, 
or even halfway through the year, is simply too late to 
prevent many students from dropping out. Moreover, 
this approach discards very valuable information for 
idenfifying how far on-frack or off-track students are 
during or after their first year of high school. It became 
clear from our work in MRS fhat a more nuanced way of 
capturing the quality of students' academic experiences 
in addition to the guantity (e.g., credit attainment) was 
needed. Additionally, it seemed that a useful on-track/ 
off-frack measure could be replicafed for school years 
both prior to and following the first year of high school, to 
both be consistent with the principle of an early warning 
system as well as to recognize the fact that many MRS 
students remain at risk despite persisting into their 
second year of high school and beyond. 

The result of this thinking is a new measure called Total 
Quality Credit (TQC) attainment, which combines credit 
attainment and final grades into a continuous variable 
that can be can be used as an early warning “shortcut" to 
predict the most valued high school outcomes (graduation 
and college enrollment) based upon middle school data, 
rather than waiting until these outcomes actually occur 
among high school students up to six years after initial 
9th grade enrollment. Under the two-stage recursive 
modeling structure, middle grades outcomes (including 
both academic inputs and engagement measures such 



4 Most of the students who are in their sixth year of high school enrollment are students with disabilities; the Individuals with Disabilities Education Act (IDEA) stipulates that school districts are 
required to provide services to students with disabilities up to the age of 21 . 
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as attendance and behavior) are used to predict TQC 
attainment at the end of students’ first year of high school, 
while first-year TQC is in turn used to predict the “final” 
outcomes of high school graduation and initial college 
enrollments 

Part 4. Combining Risk Factors into an Eariy 
Warning System 

Having investigated the descriptive relationships that 
exist between individual independent variables (such as 
academic attainment as measured by TQC, attendance, 
mobility, retention, student demographics, etc.) and the 
desired high school outcomes (graduation and initial 
college enrollment), the next step was to combine these 
risk factors into predictive models for understanding TQC 
attainment at the end of the first year of high school. To 
this end, a series of regression analyses using first-year 
TQC attainment among first-time MRS 9th graders in 
2005-06 and 2006-07 (combined) as the dependent 
variable were prepared. 

Independent (predictor) variables used in the analyses 
include the following: 

• School type from which a student “graduated” from 
8th grade (K-8 vs. traditional middle school vs. 
middle/high school)® 

• Grade 8 WKCE reading and math proficiency level 

• Absence rates during each year of the middle 
grades 

• TQC attainment during each year of the middle 
grades 

• Instances of mobility during each year of the middle 
grades (mobile vs. not mobile) 

• Retention during each year of the middle grades 
(retained vs. not retained) 

• Suspensions during each year of the middle grades 
(suspended vs. not suspended) 

• Student demographics (gender, race/ethnicity, ELL, 
special education, economic status) 



FINDINGS 

Thus far, three main inferential findings have emerged 
from our analyses, although given the interim nature of 
this report, we expect that additional findings of interest 
will emerge in the near future. These include the following: 

Finding #1. “High yieid” indicators of students 
at risk for dropping out of MPS and graduating 
with low college/workforce readiness include 
academic achievement (TQC), attendance, 
suspension, and retention/overage. 

Using regression analyses, it was possible to identify a 
selected set of “high yield" indicators to show in the MPS 
Early Warning “dashboard,” which serves as the primary 
means through which early warning research findings 
are already being operationalized within the district for 
purposes of identifying students at risk. In the case of the 
MPS early warning system, regression analysis identified 
academic achievement (TQC), attendance, suspension, 
and retention/overage as the four most important 
predictors. 

Finding #2. The ability to predict first year high 
school outcomes thus far using middle school 
indicators is somewhat limited. Among other 
implications, this suggests that the transition 
to high school presents a significant challenge 
for many middle grades students in MPS, even 
those with relatively high levels of performance 
through the middle grades. 

Another main inferential finding emerging from these 
analyses is that using even the most complete sets of 
middle school outcomes provides somewhat limited 
predictive power in terms of predicting end of first-year high 
school TQC attainment - and that having multiple years 
of middle grades outcomes, as opposed to just a single 
year (e.g., grade 8) appears to add somewhat minimal 
predictive power for predicting first-year TQC attainment. 
In other words, many MPS students who have done 
reasonably well throughout middle school in terms of 



5 It should be noted that these outcomes can be thought of as “placeholders" for now, until MPS obtains sufficient data to inform the much more important outcome of college persistence and 
graduation. 

6 This is an important consideration in MPS because the district, like several of its urban counterparts, began converting many of its traditional middle schools (serving grades 6-8) to a K-8 
configuration beginning in the late 1 990s. Accordingly, the question of whether school type (K-8 vs. middle) exerts any degree of measurable influence upon high school outcomes for students 
with similar backgrounds becomes a key policy question. 
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both academics and engagement (e.g., attendance and 
behavior) experience substantial academic deficiencies 
during their first year of high school. 

Finding #3. Irrespective of school type (K-8 vs. 
traditional middle), some MRS middle schools 
appear to be doing a better job than others in 
preparing students for the rigor of high schooi. 

The inclusion of school-level Indicators to identify both the 
middle school from which each MPS student “graduated” 
as well as the school(s) they attended during their first 
year of high school also told an interesting story. While the 
overall (districtwide) relationship between outcomes In 
the middle grades and those during the first year of high 
school was perhaps somewhat weaker than expected, 
stronger relationships emerge for some schools than for 
others. This suggests that some MPS middle schools 
are doing a better job than others in preparing students 
for the rigor of high school. It may further suggest that 
having the right “match” of graduating 8th grade student 
to high school acts as a critical determinant of success 
at the high school level and beyond, In much the same 
way that the matching process of high school graduate 
to college has been shown to be very important as a 
determinant of college success. 

The impact of school type and other variables of potential 
interest are also discussed at the end of this section. 

CONCLUSION 

While the research component of the MPS early warning 
system conducted by VARC continues, the district has 
already taken steps to incorporate early warning findings 
into “products” intended to inform district improvement 
efforts. MPS has developed an “early warning dashboard” 
at the high school level, for example, that provides up- 
to-date information for the four “high yield” indicators 
identified (TQC, attendance, suspensions, and refention/ 
overage), with specific thresholds set for each year in high 
school that place a student into one of three risk categories: 
green (low risk), yellow (moderate risk), and red (high risk). 



In addition, several policy implications and avenues for 
future work have emerged from VARC's early warning 
work In MPS. This work has contribufed to a deeper 
understanding of the dropout phenomenon in MPS, 
and by knowing more about dropouts in a descriptive 
sense- such as when they drop out most frequently 
during their high school career - it is possible to design 
interventions to more effectively address this problem. 
For example, the fact that many MPS dropouts persist 
Into well into their high school careers suggests that 
dropping out Is best understood as a gradual process of 
disengagement rather than a single catastrophic event, 
and that efforts to better identify af-risk sfudents based 
on observable outcomes during the middle grades and 
even the first year of high school are critical. At the same 
time, the fact that the modal grade for MPS dropouts 
Is grade 9 suggests that additional programming 
targeted at easing the difficult transition from middle 
to high school Is warranted, and that this programming 
should be informed by rigorous early warning data. 

While early warning work conducted to date has been 
very promising, another policy implication emerging from 
our work is thaf additional research and development is 
clearly needed to “fine tune” the system. For example, 
our analysis illustrates that It is misleading to think 
of graduates as a single, monolithic group whose 
trajectories for life success are largely similar. Instead, 
we identify a second key outcome that should be 
predicted by early warning work, in addition to predicting 
which students are at elevated risk of dropping out: 
those students who are likely to graduate, but with 
very low levels of college and workforce readiness. 

Moreover, the low predictive power of middle grades 
models suggests that additional data are needed In order 
to make these early warning models more robust. This may 
include developing ways of measuring key non-cognitive 
outcomes such as students' sense of attachment to school 
and the efficacy of their support networks, and perhaps 
also incorporating non-school predictive information such 
as students' health and family backgrounds. 
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OVERVIEW OF PRIOR RESEARCH AND 
DEVELOPMENT EFFORTS 

Early warning system development in MRS has benefited 
greatly from prior research in this area. A general 
consensus has emerged from prior research (Jerald, 
2006; Heppen & Therriault, 2008, Kennelly & Monrad, 
2007; Balfanz & Herzog, 2005) that three broad types 
of factors put students at greater risk of dropping out of 
school: 

• Social background: research clearly shows that 
students who have certain characteristics - includ- 
ing being from low-income, single-parent, and 
less-educated families, and taking on adult respon- 
sibilities such as becoming a parent and needing to 
work - are more likely to drop out. 

• Students’ educational experiences: 

0 Academic performance: students who fall 
behind academically, as shown by low grades 
and test scores, and by being retained at 
grade level, exhibit substantially higher drop- 
out rates 

0 Engagement in school: students with high 
rates of absenteeism and poor behavior, and 
lower rates of participation in school activi- 
ties, have been shown to exhibit higher rates 
of dropout 

• School characteristics: various combinations of 
factors including perceptions of school climate and 
teacher support, school size, and rigorous curricula 
with high expectations have been shown to exert a 
substantial influence on dropout rates. 

With respect to using the social background factors of 
individual students as predictors of dropout likelihood, 
Jerald (2006) notes that until quite recently, these were 
the primary criteria used to identify students at the 
greatest risk of dropping out. More in-depth research 
revealed at least three major problems with this approach, 
however. A first is that it was never clear how many, and 
which combinations, of risk factors needed to be present 
in order to place a student at high risk of dropping out, nor 
at which ages the various combinations of risk factors held 
the greatest predictive power. As just one example, while 



more instances of mobility (e.g., school changes) across 
a student's entire educational career might be associated 
with increased dropout probability, at least one study 
(Swanson & Schneider, 1 999) found that mobility during 
the early part of high school can actually be beneficial, 
to a point, in terms of reducing dropout likelihood, in that 
leaving a high school in which a student was not having 
a good experience was preferable to remaining there. A 
second problem is that focusing on social background, or 
"status" (e.g., unchangeable) indicators as determinants 
of dropout likelihood left schools with little that was 
"actionable” on their part: what could a school do, after all, 
to change the fact that a student was from a low-income 
or single-parent family? 

A third problem that became evident from the research is 
that focusing on background/status indicators deflected 
attention in many cases from other categories of risk 
factors noted above, which have ultimately proved to be 
much more important, or "high yield,” indicators. In other 
words, it is not so much the case that being from a low- 
income or single-parent family per se causes a student 
to be at risk of dropping out, but rather his/her lack of 
academic progress, and/or lack of engagement In 
school, that produces this outcome. The two tend to be 
highly correlated, of course, but the point is that focusing 
on the factors that are actionable on the part of schools 
(e.g., whether students are progressing academically, and 
whether they view school as important and welcoming) is 
more accurate, in that it allows schools to most effectively 
and efficiently direct resources toward students who 
display “real” and accurate warning signs rather than 
targeting students who fit a certain “profile” or “checklist” 
of presumed risk categories, but are not in fact at elevated 
risk of dropping out because their academic progress is 
sufficient. Accordingly, the previously-accepted method 
of targeting prospective dropouts based on “checklists” 
of social and demographic risk factors was shown to 
be inefficient, because this approach not only targeted 
scarce resources toward students who in many cases 
did not wind up dropping out (e.g., the “Type 2” error, or 
the “false positive” problem), but also because it wound 
up missing many students who did drop out (Gleason & 
Dynarski, 2002). 
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With this background in mind, recent efforts to develop 
early warning systems have focused primarily on "mining” 
data sets for student-level information that points to 
elevated risk of adverse outcomes such as dropping out. 
Work conducted in Chicago and Philadelphia provides 
useful and well-documented examples here. In Chicago, 
the Consortium on Chicago School Research (CCSR) 
has conducted extensive work identifying risk factors that 
correlate strongly with graduation among students in the 
Chicago Public Schools (CPS), and arrived at four non- 
status measures: attendance, GPA, credit attainment, and 
failing grades (Allensworth & Eason, 2005). In particular, 
the combination of the latter two elements into an “on 
track” indicator for CPS 9th graders proved to be highly 
predictive of graduation, as more than 80% of students 
who are on-track wind at the end of grade 9 wind up 
graduating, compared to only 20% graduation probability 
for those who are off-track at the end of 9th grade. More 
detailed discussion of the on-track indicator developed 
in CPS, along with an adaptation for use in Milwaukee 
that we have called “Total Quality Credits,” appears in 
the methodology section of this paper. In Philadelphia, 
University of Pennsylvania and Johns Hopkins University 
researchers similarly found a student's first year of high 
school to be a “make or break” year that was highly 
predictive of graduation probability, but also delved into 
students’ middle school records to find that grades 
(including GPA and course failures, most notably) and 
attendance were highly correlated with the probability of 
graduating (Balfanz & Herzog, 2005). 

Emerging from work in Philadelphia, Chicago, and other 
districts, therefore, was a growing consensus that early 
warning systems for identifying students at high risk 
of dropping out should focus on two main areas of 
data that are readily available in most cases: academic 
achievement {\nc\ud\ng grades obtained in core courses, 
course failures, and retention) and student engagement 
(primarily attendance and suspension data). Interestingly, 
and perhaps somewhat surprisingly, work in both 
districts found that while these middle school indicators 
served as important predictors of graduation probability, 
standardized test scores from middle school were, in 
general, not nearly as useful as predictors of adverse 
outcomes in high school. On the one hand, Chicago 



students with high 8th grade test scores were indeed 
more likely to be “on track” for graduation at the end of 
9th grade, but it was also the case that many students 
with high test scores in middle school experienced 
significant difficulties after starting high school, while 
(vice versa) many students with comparatively low middle 
school test scores did fine as 9th graders. This same 
finding, as will be seen, emerges from our work in MPS. 
One explanation offered by Balfanz and Herzog (2005) 
for this apparent contradiction is that high schools 
themselves play a substantial role in determining whether, 
and which, students stay on track, and that unfortunately 
the transition to 9th grade acts like “a kind of quicksand" 
for many students, in that risk factors not observed prior 
to high school suddenly manifest themselves upon entry 
into 9th grade. 

Having identified from prior research a set of “high yield" 
indicators upon which to base the initial phase of building 
an early warning system, a key, related question raised in 
the Philadelphia and Chicago work involves how far back 
in a student’s educational history researchers should go 
in developing the most precise early warning indicators. 
Despite claims occasionally seen in research and 
commentary, there appears to be little thus far in terms of 
rigorous, compelling research to suggest that it is possible 
to identify, with a high degree of accuracy, students who 
are most at risk of failing to graduate based on outcomes 
as early as third grade. Instead, there appears to be 
some consensus emerging from early warning work (see 
Roderick, 1993; Roderick & Camburn, 1999; Roderick, 
Allensworth, & Nagaoka, 2004; Kennelly & Monrad, 
2007) that the beginning of middle school is the optimal 
“starting point" for establishing the most accurate and 
precise early warning models. While in some cases it 
may be useful to incorporate pre-middle school data (e.g., 
from the elementary grades K-5), Roderick (1993) and 
others report that for many students, the patterns which 
ultimately predict dropouts with the highest degree of 
precision have not been sufficiently developed prior to 
middle school. Nearly two-thirds of eventual dropouts 
in Roderick's Massachusetts study, for example, had 
4th grade attendance and grade patterns that were 
not appreciably different from students who wound up 
graduating in the bottom third of their class. 
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In addition, using outcome data from elementary school 
may is thought to be somewhat premature in the sense 
that key “transitions” (e.g., from elementary to middle 
school, and from middle to high school) that are hugely 
significant in students’ decisions to drop out have not yet 
occurred. Finally, it is typically the case that rich sets of 
outcome data are much harder to obtain for elementary 
(and to a lesser extent, middle school) students than is 
generally the case for high school students. While many 
student information systems store attendance and test 
score information for elementary students, for example, 
it is less common to have final grades and other useful 
input measures such as behavioral incidents for younger 
students, particularly as one goes further back in time. 
This was certainly the case with our early warning work 
in MRS, as the question of how far back in time to go 
essentially resolved itself in the sense that very limited 
outcome data from elementary grades was maintained in 
a reliable and consistent manner prior to 2005. The good 
news here, of course, is that prior research indicates that 
this is likely not a significant limitation, in that focusing on 
outcomes from middle school onward produces the most 
useful results. 

No early warning system, of course, can possibly 
identify 100% of students who eventually drop out, or 
graduate with low levels of college readiness, since a 
host of school-specific and student-specific factors 
(some of which can't be readily tracked using data that 
schools typically collect and have access to) are known 
to influence dropout decisions. The Consortium on 
Chicago School Research, for example, has consistently 
found substantial differences across Chicago high 
schools - even those that are very similar in terms of 
size, student demographics, and other characteristics - 
with respect to having a “culture” that supports students’ 
sense of engagement and connection with their school. 
Accordingly, the Consortium has greatly increased efforts 
to measure school culture through surveys and focus 
groups with students. As noted later, this represents an 
important direction for future research on early warning 
systems in M PS, as the data available in the development 
of this first early warning system was not nearly as 
comprehensive as those available in other districts. For 
example, a research consortium in Philadelphia has 



populated a database called the Kids Integrated Data 
System (KIDS) with not just school-based variables 
(student demographics, grades, attendance, test scores, 
etc.), but also a host of non-school variables related to 
students’ lives and histories, such as parental income and 
educational attainment, social services data (incidents 
in which child protective services incidences), and 
health-related indicators such as exposure to lead paint 
Research in Philadelphia using the KIDS database has 
shown that some of these non-school indicators, in fact, 
have even greater predictive power in terms of identifying 
future dropouts than do school-based indicators (Rouse 
& Fantuzzo, 2009). While comparable data-sharing 
agreements are not yet in place in Milwaukee, the point 
here is to emphasize that the early warning work in MPS 
summarized in this paper represents only a first step in 
what is hopefully a much longer-term effort to incorporate 
much richer data sets for identifying students at risk of 
adverse outcomes. 

Very recently, and following the pioneering early warning 
work in Chicago and Philadelphia, a number of studies 
(see, for example, Williams, Kirst, Haertel, et al. [2010], 
Kurlaender, Reardon, ef al. [2008] and Zau & Betts 
[2008]) have corroborated earlier findings which showed 
that many students at the greatest risk of failure in high 
school could in many cases be identified based upon 
combinations of middle school outcomes that generally 
include grades, attendance, behavior, and test scores. 
This research further demonstrated that success in the 
middle grades is a strong predictor of success in high 
school and beyond, including high school grades and 
credit attainment, scores on high school exams (of both 
an NCLB and end-of-course variety), and probability of 
enrolling in college. At the same time, however, each of 
these studies also corroborated, to varying degrees, the 
finding from Chicago and Philadelphia research that even 
the best sets of middle school data had limitations in the 
extent to which they were able to accurately identify 
future dropouts. In other words, the troubling conclusion 
remains that a very unfortunately, dismal process sets 
in when many students enter high school, culminating 
in these students eventually dropping out, that is not 
generally revealed by even the richest sets of middle 
school outcome data. 



18 



The Council of the Great City Schools 





Finally, while early warning systems developed to date 
have, quite appropriately focused on identifying students 
at risk for not completing high school, it became clear 
early on in the Senior Urban Fellowship work that a 
parallel area of concern was equally important. This 
involves students whose academic performance and level 
of engagement in school make them reasonably likely 
to graduate, but with very little evidence of having skills 
that signify readiness for college and the workforce. As 
such, the early warning system as developed for use in 
MRS very much has a dual purpose: to identify students at 
risk of dropping out, but also to identify and intervene on 
behalf of students for whom graduation is a reasonably 
likely outcome, but prospects for college and workforce 
participation and success are much less positive. 

There is ample evidence that including a focus on college 
and workforce readiness, in addition to graduation 
probability, is a worthwhile goal tor an early warning 
system. On a national level, as seen above, some good 
news clearly exists in that there has been slow, steady 
improvement in the percentage of young Americans 
(ages 18-24) who hold high school equivalency 
credentials. At the same time, however, there is concern 
that the rigor of many students' high school experiences 
may have been sacrificed in the process. Data from the 
NAEP High School Transcript Study (HSTS), for example, 
show that graduates from the class of 2005 completed 
more credits in high school (nearly three more, in fact), 
and earned higher grades (approximately one-third of a 
letter grade) compared with their counterparts from the 
class of 1 990 (Shettle et al., 2007). It is also true that 
the percentage of U.S. high school graduates who attend 
college immediately after graduating has increased 
steadily over the past three decades, to the point where 
two-thirds of graduates now enroll in college right after 
they finish high school (Knapp, Kelly-Reid, & Ginder, 
2010 ). 



At the same time, however, the scale score performance 
of 12th graders nationwide on NAEP reading 
assessments has actually declined somewhat over the 
past two decades,^ and the percentage of students who 
attain “college ready” scores on ACT subject tests® has 
remained virtually unchanged over at least the past five 
years. While two-thirds of 2008 high school graduates 
nationwide who took the ACT and went on to college 
reached the college readiness benchmark for English 
(whose corresponding college-level course is freshman- 
level English Composition), this is virtually identical to the 
level attained by 2004 graduates. Only 53% of 2008 
graduates reached the college readiness benchmark for 
Reading (predicting readiness for freshman-level Social 
Science), 43% reached the Math benchmark score 
(predicting readiness for college Algebra), and 28% 
reached the Science benchmark (predicting readiness 
tor college Biology). In all instances, the percentage of 
2008 graduates who attained ACT college readiness 
benchmarks was virtually unchanged from 2004 (ACT, 
2009). 

In addition to the national-level data noted above, multiple 
sources of MPS-specific data validate the inclusion of 
college and workforce readiness indicators in an early 
warning system. While the percentage of MPS graduates 
who have completed rigorous coursework (defined as four 
years of English and three years each of math, science, 
and social studies) has increased from 38% for 2007-08 
graduates to 48% for 2008-09 graduates, for example, 
both figures fell below goals set forth in the district's 
strategic plan, and there has been no corresponding 
increase in either 10th grade WKCE performance nor 
ACT composite scores to serve as an external validation 
of this apparent increase in course-taking rigor at the 
high school level. ACT subject-area sub-scores show, 
furthermore, that very few MPS students obtain scores 
that signify college readiness. Among MPS 1 1th graders 



7 The mean national NAEP reading score for 1 2th graders in 2005 was 286, which is down from 292 in 1 992. Since testing accommodations were not allowed for the 1 992 administration, but 
were allowed in 2005, perhaps a better comparison is between 1 998 results (mean scale score = 290) and 2005 (mean scale score = 286). NAEP mathematics performance data for 1 2th 
graders in 2005, the most recent year of administration for this age group, are not comparable to prior years due to a re-scaling of the 1 2th grade math test 

8 College readiness benchmark scores as defined by ACT research are the thresholds on ACT subscores in Reading, English, math, and science that are associated with at least a 50% 
probability of obtaining a grade of B or better, or a 75% probability of obtaining a C or better, in corresponding freshman-level college courses. 
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who took the ACT in 2009, for example, only 3% had 
scores that met all four ACT benchmark scores, and only 
6% and 4% had ACT Science and ACT Math scores 
that met college readiness benchmarks, respectively.® 
Data reported In the MRS Strategic Plan, additionally, 
Indicate that nearly three-fourths (72.7%) of MRS 2008- 
09 graduates who went on to attend the University of 
Wisconsin-Milwaukee required remedial coursework as 
college freshmen, while 23.0% of MRS graduates who 
attend Milwaukee Area Technical College are placed 
into Basic Skills programs, rather than the more desired 
placement in occupational programs. As a final example, 
and as discussed in more detail in the next chapter, 
MRS high schools produce each year a large number 
of graduates with extremely low grade-point averages 
(e.g., around or below 1.0), which makes it very difficult 
to either continue their education at the college level or 
participate meaningfully in fhe workforce. 



9 ACT college readiness benchmark scores are the minimum scores, validated by ACT research, needed on ACT subject-area tests (English, Math, Reading, and Science) to indicate a 50% 
chance of obtaining a B or higher or a 75% chance of obtaining a C or higher in the corresponding credit-bearing college course. 
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PART 1. OVERVIEW OF DATA USED 
AND CHALLENGES ENCOUNTERED 



This section describes the sources and availability of data 
that have been used in building the early warning system 
in MPS, along with challenges encountered during 
this process. Given the importance of fhe first year ot 
high school in terms of predicting eventual graduation/ 
dropout outcomes, as established through research in 
Chicago and Philadelphia, it makes sense in terms of 
naming and labeling conventions to use this year (first 
time grade 9 enrollment) as the “starting point” in an early 
warning system, and to work backward (e.g., into middle 
school) and torward (into high school) from there. 

At the onset of the early warning work in MPS, therefore, 
the goal in terms of assembling data sets was to build a 
comprehensive “master file” containing all relevant data 
to encompass the educational experiences ot several 
cohorts of MPS students starting with first-time 9th grade 
enrollment. Since an obvious goal of an early warning 
system is to identity students at elevated risk ot adverse 
outcomes as soon as possible - but not too soon so as 
to be misleading - the research noted previously from 
Chicago and Philadelphia suggesfed fhat middle school 
(e.g., fhe beginning of grade 6) represented a suitable 
“starting point” tor describing the educational experiences 
ot tirst-time 9th graders. As far as the “ending point” for 
each cohort ot first-time 9th graders, an obvious choice 
is the high school “final outcome” (e.g., graduation or 
dropping out), although even more desirable would be 
data documenting post-high school outcomes such as 
initial college enrollment and graduation, along with wage 
data. 

Upon turther consideration, assembling a data set to 
describe the full educational experience of first-time 
9th graders turns out to be a rather ambitious goal, as it 



requires at least 9 years of panel dafa for the same cohort 
to accurately describe just the middle and high school 
portion of these students' educational experiences, as 
follows: 

• Four years to capture the full middle school experi- 
ence, from entry as first-time 6th graders (three 
years of schooling, plus af least one additional year 
to capture one instance of retenfion during the 
middle grades) 

• Six years to capture the full high school experience, 
from entry as first-time 9th graders (four years of 
schooling in high school, plus fwo additional years 
to capture two instances of retention during grades 
9-12.'“ 

The reality of the available data in MPS tell somewhat 
short of this ideal data scenario. To begin with, complete 
middle school data sets (containing students' tinal grades 
and courses taken, test scores, monthly attendance rates, 
behavioral incidents, etc.) were available only from the 

2000- 01 school year onward, for example, which meant 
that final high school outcomes (graduation vs. dropout) 
could not be verified for first-time 6th graders in 2000- 
01 until 2010-1 1 at the earliest. In other words, under 
the complete, 9-year panel data model, we are etfectively 
limited to looking at middle school outcomes tor students 
who were tirst-time 9th graders no earlier than 2004- 
05, since these students would have been 6th graders in 

2001- 02 (assuming they didn’t repeat any grades during 
middle school). First-time 9th graders in 2004-05 (and 
2005-06, recalling that multiple cohorts are preferable 
for generalizability), however, would not graduate under a 
full, six-year portrait of high school" until 2009-10 and 
2010-1 1 , respectively. 



1 0 The annual MPS retention rate for grade 9 alone Is approximately 25-30%. 

1 1 According to a 2008 MPS study of high school completion patterns, six years is necessary to most accurately determine whether students graduate or drop out as their final outcome, due to 
high rates of retention in MPS at the high school level, particularly in grade 9. 
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FIGURE 1: TOTAL QUALITY CREDIT (TQC) ATTAINMENT LOGIC MODEL WITHIN EARLY WARNING SYSTEM 
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A related problem was that MPS’s eSIS system, which 
Is the district's student-level transactional data system 
documenting student enrollment, course-taking histories, 
and other data elements necessary for building an early 
warning system was not implemented in the middle 
grades until the 2005-06 school year. 

A final issue was that MPS operated two separate data 
warehouses for a time during the transition to a new 
system which became operational in 2006-07. Certain 
data elements were supplied to both the new and old 
systems for several years, but others - including some 
aspects of course-taking histories and final marks - were 
available for some years in either the old or new system, 
but not for all years in both systems. 

In balancing the data needs of an effective early warning 
system with the availability of MPS data required for 
this purpose, therefore, it was obvious at an early phase 
of work that a “workaround" solution was necessary. 
The solution, broadly stated, was to employ a two- 
stage recursive data modeling structure'^ that provides 
a “complete” data set by linking together necessary 
elements from disparate sources. 



More specifically, the recursive structure used in our 
work combines necessary data elements for the middle 
school portion of students' educational experience from 
two distinct cohorts of students (first-time 9th graders 
in 2005-06 and 2006-07) with different data sets 
describing the high school portion of the educational 
experience (from two different cohorts of first-time 9th 
graders, in 2001-02 and 2002-03). These two data 
sets are linked together through an outcome measure 
developed for this project that we have entitled “Total 
Quality Credits" (TQC) during the first year of high school, 
which is a construcf (described more fully later on in this 
chapter) that combines credit attainment and final grades 
to create a very useful measure of academic progress. In 
other words, what the two-stage recursive model does is 
to predict end of high school outcomes (graduation and 
initial college enrollment) using TQC attainment during 
the first year of high school, which is itself predicted 
by a full set of outcome data (grades, attendance, test 
scores, etc.) from middle school. This process is depicted 
in Figure 1 above. 



1 2 See Gruenewald et al, 2008 
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PART 1. OVERVIEW OF DATA USED 
AND CHALLENGES ENCOUNTERED 



The first step in the recursive data modeling process is 
to determine which students are first-time 9th graders, 
which is considered year t. This was done using third 
Friday enrollment counts for each year, with a student 
considered to have initial grade 9 enrollment in a given 
year if they are in grade 9 that year, but not in grade 9 the 
prior year (either because they are new to M PS for grade 
9 in year t or, in most cases, because they were in grade 
8 the prior year). The importance of using chronological 
year in school (e.g., year t, t-1 , t-l-1 , etc.), rather than grade, 
as the “yardstick” for measuring student progress due to 
the fact that the MPS retention rate for grade 9 has been 
in the 25-30% range in recent years. Under this logic, the 
naming structure for each year is thus as follows: 

• Year t-l-5: sixth year of high school 

• Year t-l-4: fifth year of high school 

• Year t-l-3: fourth year of high school (grade 1 2 
under normal grade progression) 

• Year t-l-2: third year of high school (grade 1 1 under 
normal grade progression) 

• Year t-l-1 : second year of high school (grade 1 0 
under normal grade progression) 

• Year t: first year of grade 9 

• Year t-1 : grade 8 

• Yeart-2 (grade 7 under normal grade progression) 

• Year t-3 (grade 6 under normal grade progression) 



To summarize the data modeling structure, our strategy 
for developing the early warning system for MPS has 
been to incorporate two distinct sets of data; 

• Identifying risk factors based on students’ experi- 
ences during the first year of high school and 
beyond, using known dropouts and graduates'^ 
from two cohorts of first-time MPS 9th graders 
(2001-02 and 2002-03); 

• Identifying a complementary set of risk factors 
based on students' experiences during middle 
school, using data from two separate cohorts of 
first-time MPS 9th graders (2005-06 and 2006- 
07). 

While having a full, 9-l-year set of panel data for the 
same cohort of students is perhaps a “cleaner" approach 
toward building an early warning system, it turns out that 
using the two-pronged data approach described above 
(e.g., splicing together high school outcomes using first- 
time 9th graders in 2001-02 and 2002-03 with middle 
school outcomes using first-time 9th graders in 2004-05 
and 2005-06) does otter at least one major advantage: 
much more current data can be used. In other words, 
using the full panel of data for the same cohorts of 
students essentially supports conclusions about middle 
school risk factors that is somewhat “old” data by the time 
findings can be generated. That is, just because middle 
school risk factors from 2003-04 were good predictors 
of graduation does not necessarily mean that the same 
set of middle school risk factors from 2007-08 were 
equally good predictors of graduation, as the “landscape” 
of middle school might have changed in important ways 
(e.g., a revamping of the middle school reading curriculum 
or disciplinary policies, moving from traditional middle 
schools to K-8 arrangements, etc.). 



1 3 The process for determining which students are “confirmed" dropouts involves a set of codes used in the MPS electronic student information system (ESIS) to ensure that a student is only 
enrolled in one school at a time. When a student leaves a school, a code is entered to describe his/her reason for leaving and next destination, if known (e.g., graduated, left for another school 
within MPS, left for another district in Wisconsin, left to go out of state, incarcerated, deceased, etc.). In the end, a student Is counted as a dropout if they have one of three codes that represent 
what is “left over" after all other possible outcomes have been accounted for. Due to the fact that a considerable number of MPS students who wind up graduating take more than four years to 
do so, “graduates" are considered to be any students from the cohorts of first-time 9th graders in 2001-02 and 2002-03 that graduated within six years. 
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PART 2. DATA INVENTORY: 

WHAT IS KNOWN ABOUT MPS DROPOUTS? 



A logical next step in building an early warning system 
is to assemble descriptive data on potential “high yield” 
indicators identified in prior research as being predictive 
of adverse outcomes such as dropping out and graduating 
with low levels of college/workforce readiness. The 
following section summarizes key descriptive findings 
related to known dropouts from two cohorts of first-time 
MPS 9th grade students (in 2001-02 and 2002-03), 
based upon a longitudinal cohort study conducted by the 
MPS Division of Research and Assessment 

• How early into their high school career do most 
MPS students tend to drop out? 

A key piece of information underlying the development 
of an early warning system is to get a sense of when 
students in MPS tend to drop out most often, both in 
terms of how many years into high school as well as 
students' last grade level prior to dropping out 



Table 1 shows dropout frequency data by both length 
of time after first-time grade 9 enrollment and by last 
known grade of enrollment among first-time MPS 9th 
graders in 2001-02 and 2002-03. The information 
used to determine when a student drops out comes 
from “withdrawal codes” taken from the district's ESIS 
database, as described in Footnote 1 3. It is acknowledged 
here that using dates associated with ESIS withdrawal 
codes likely has some degree of imprecision in terms 
of accurately capturing the specific date that a student 
quit coming to school long enough to be classified as a 
dropout, but this remains the best source of information 
that can be used for this purpose. 

An initial observation is that a relatively small number of 
MPS students who wound up as dropouts (7.1% across 
the two cohorts combined) did so during their first year 
of high school. This figure closely matches the 5% of 



TABLE 1: DROPOUT FREQUENCY FOR FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 BY 
LENGTH OF TIME (IN YEARS) AFTER INITIAL 9TH GRADE ENROLLMENT AND LAST GRADE 



NUMBER AND % OF DROPOUTS BY COHORT OF 
FIRST-TIME 9TH GRADERS: 




2001-02 


2002-03 


2-YEAR TOTAL 


Dropout Frequency by Length of Time 
after Initial 9th Grade Enrollment: 








During 1 st year of high school (year t) 


69 (5.7%) 


1 1 1 (8.3%) 


180 (7.1%) 


During 2nd year of high school (year t+1 ) 


200 (1 6.4%) 


172(12.9%) 


372 (14.6%) 


During 3rd year of high school (year t-l-2) 


254 (20.9%) 


313(23.5%) 


567 (22.3%) 


During 4th year of high school (year t-l-3) 


357 (29.3%) 


391 (29.4%) 


748 (29.4%) 


During 5th year of high school (year t-r4) 


237(19.5%) 


247 (18.6%) 


484(19.0%) 


During 6th year of high school (year t-l-5) 


74(6.1%) 


96 (7.2%) 


1 70 (6.7%) 


Total Dropouts 


1218(100.0%) 


1330(100.0%) 


2548(100.0%) 


Dropout Frequency by Last Grade of 
Enrollment: 








9 


264 (21.7%) 


398 (29.9%) 


662 (26.0%) 


10 


289 (23.7%) 


345 (25.9%) 


634 (24.9%) 


1 1 


292 (24.0%) 


330 (24.8%) 


622 (24.4%) 


12 


149(12.2%) 


169 (12.7%) 


318(12.5%) 


Other grades/missing data''* 


224(18.4%) 


88 (6.6%) 


312 (12.2%) 


Total dropouts 


1218(100.0%) 


1330(100.0%) 


2548(100.0%) 



1 4 In some cases, there is no last grade recorded for students who have one of the three exit codes used to denote dropouts, while in another (very small) number of cases, the last recorded grade 
represents an obvious data entry error (e.g., grade 3). A third group of cases has a last grade of 8; this may represent students who reached the age of 1 8 (the earliest age at which a student 
can legally drop out in Wisconsin under current state law), but were still in 8th grade at the time they dropped out due to multiple instances of repeating a grade. This third group of students is 
included in the analysis of when “first-time 9th graders” in 2002-03 dropped out, even though they never actually reached grade 9, because It is important to get the most accurate picture of 
when students drop out. 
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PART 2. DATA INVENTORY: 

WHAT IS KNOWN ABOUT MPS DROPOUTS? 



students who drop out ot Philadelphia high schools during 
their tirst year ot high school (Neild & Farley, 2004). It is 
more common for MPS students, as was true for their 
peers in Philadelphia, to drop out during their third and 
fourth years of high school, although a surprising number 
of MPS students make it to their fifth and even sixth'^ 
year of high school before dropping out. This observation 
is corroborated by the fact that MPS dropout rates by 
grade show that the highest "event” dropout rate for any 
of the high school grades in 2008-09 was at grade 1 1 
(10.4%); traditionally, the highest dropout rates in the 
district had been observed for 9th graders, but in 2008- 
09 the grade 9 dropout rate of 9.6% was eclipsed by 
1 1th graders. 

As has been observed in other cities, therefore, it 
appears that dropping out of MPS as an outcome is 
best understood as a gradual process of disengagement 
and academic shortcoming that often plays out across 
multiple years of a student’s high school career, rather 
than as a single catastrophic event that leads to 
immediate withdrawal from school. 

Despite the fact that many MPS students persist into their 
fourth, fifth, and even sixth year of high school before 
dropping out, however, the largest group of dropouts in 
terms of students’ last grade prior to dropping out was 
9th graders, who were 26.0% of all dropouts for the two 
cohorts combined. This indicates that many students are 
retained in 9th grade once, and even twice in a substantial 
number of cases, before dropping out. This confirms, as 
noted previously, the need to treat grade in high school as 
a completely different variable than amount of time spent 
in high school, since so many MPS students do not make 
the “typical” progression (e.g., being promoted from grade 
9 after their first year of high school to grade 1 0, then to 
grade 1 1 at the end of their second year, etc.). 



The data also show that quite a few studenfs “linger” for 
some fime in MPS high schools - into their fifth and even 
sixth'^ years - and progress credit-wise to the point of 
attaining junior and even senior status before dropping 
out This finding is consistent with research conducted 
by Roderick (1993), who identified two distinct groups 
of dropouts: an “early” group that left school between the 
7th and 9th grades, and a “later” group that left between 
the 1 0th and 1 2th grades. 

• How do eventual dropouts and graduates differ 
with respect to key outcome indicators during the 
first year of high school and middle school years? 

The first year of a student’s high school experience is 
widely recognized, based on prior research (Allensworth 
& Easton, 2005; Balfanz & Herzog, 2005; Jerald, 2006), 
as a critical determinant of whether s/he ultimately 
graduates or drops out. Accordingly, presented below are 
a set of tables that differentiate graduates from dropouts 
among first-time MPS 9th graders in 2001-02 and 
2002-03 with respect to the following indicators: 

• GPA (unweighted)'® and passing rates in core 
subjects (math, reading, science, and social studies) 

• Grade 9 retention rates 

• Absenteeism 

• Behavioral incidents (suspensions) 

• Grade 8 test scores 

• Student mobility 



1 5 Most of the students who are in their sixth year of high school enrollment are students with disabilities; the Individuals with Disabilities Education Act (IDEA) stipulates that school districts are 
required to provide services to students with disabilities up to the age of 21 . 

1 6 Unweighted GPA is used for analysis purposes here (A=4.0, B=3.0, C=2.0, D=1 .0) because MPS students' access to Advanced Placement courses Is quite uneven across high schools; many 
sites offer few, If any, such courses. 
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GPA 



Starting with mean GPA in the four “core” academic 
subjects of reading, math, science, and social studies, 
it is not surprising to find that eventual dropouts had 
substantially lower first-year GPAs in the core subject 
areas of math, reading, science, and social studies 
courses than did eventual graduates. Note here that 
“graduates” include all students who graduated within 
six years of initial grade 9 enrollment, rather than just 
students who graduated on time; this decision was 
made given the high rates of grade 9 retention in MPS 
noted above, which preclude many MPS students who 
eventually do graduate from doing so on-time. As seen in 
Table 2, there is massive failure among eventual dropouts 
with respect to core academic courses. More than 90% 
of first-time 9th graders in the two cohorts combined 
who wound up dropping out (n=2548) had a mean GPA 
across their first-year high school math courses of 1 .0 or 
lower, and fully 60% of math courses taken by eventual 
dropouts resulted in a failing grade. The performance of 
eventual dropouts in reading, science, and social studies 
was only marginally better, with mean GPA of 0.76 - 0.82 
and failure rates of approximately 55%. 



By contrast, students who eventually graduated (n=7452) 
had mean core-subject GPAs around 2.0, and passed 
about 85% of their courses. 

Another, more visual depiction of the relationship between 
course outcomes and graduation probability is seen in 
Figure 2, which shows graduation rates by overall first- 
year GPA among first-time 9th graders in 2001-02 and 
2002-03 combined. Importantly, there is relatively little 
decline in graduation probability from an A average (4.0 
GPA) to approximately a C average (2.0), although the 
odds of graduating decrease markedly below a C/2.0. 

The very low GPA and passing rates among eventual 
dropouts are even more discouraging when considering 
that numerous students drop out early enough during 
their first year of high school (e.g., before the end of fall 
semester) that they have not yet received any final marks. 
If all of these students had Instead remained enrolled 
through the end of the fall semester, when they would 
have received their (presumably very low) semester 
grades, the mean GPA and passing rates among dropouts 
would likely be even lower. 



FIGURE 2: GRADUATION RATE BY FIRST-YEAR OVERALL GPA FOR FIRST-TIME MPS 9TH GRADERS IN 
2001-02 AND 2002-03 (COMBINED) 
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TABLE 2: COMPARISON OF FIRST-YEAR GPA IN CORE ACADEMIC SUBJECTS FOR KNOWN DROPOUTS 
AND GRADUATES AMONG FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 



200 


■02: 


2002-03; 


2-YEAR TOTAL (COMBINED): 


DROPOUTS 

(N=1218) 


GRADUATES 

(N=3762) 


DROPOUTS 

(N=1330) 


GRADUATES 

(N=3690) 


DROPOUTS 

(N=2548) 


GRADUATES 

(N=7452) 


GPA in first-year Math courses: 














0.00-1.00 


75.1% 


28.3% 


75.5% 


29.0% 


75.3% 


28.6% 


1.01-2.00 


1 5.2% 


27.8% 


1 5.4% 


28.7% 


1 5.3% 


28.3% 


2.01-3.00 


7.6% 


25.2% 


7.4% 


23.3% 


7.5% 


24.2% 


3.01-4.00 


2.0% 


1 8.7% 


1 .8% 


1 9.0% 


1 .9% 


1 8.9% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


Mean GPA for first-year math courses 


0.74 


2.04 


0.76 


2.03 


0.75 


2.04 


Pass rate tor first-year math courses 


38.5% 


84.1% 


41.2% 


84.7% 


39.9% 


84.4% 
















GPA in first-year Reading courses: 














0.00-1.00 


74.7% 


22.0% 


70.8% 


21.5% 


72.6% 


21.8% 


1.01-2.00 


1 6.8% 


30.5% 


1 9.9% 


29.9% 


1 8.5% 


30.2% 


2.01-3.00 


6.8% 


29.8% 


7.3% 


30.0% 


7.0% 


29.8% 


3.01-4.00 


1.8% 


1 7.6% 


2.0% 


1 8.8% 


1 .9% 


1 8.2% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


Mean GPA for first-year reading courses 


0.75 


2.15 


0.85 


2.19 


0.81 


2.17 


Pass rate for first-year reading courses 


41.3% 


88.6% 


47.3% 


89.5% 


44.4% 


89.0% 
















GPA in first-year Science courses: 














0.00-1.00 


77.1% 


25.0% 


72.4% 


20.9% 


74.6% 


23.0% 


1.01-2.00 


1 4.4% 


29.5% 


1 9.0% 


28.6% 


1 6.8% 


29.1% 


2.01-3.00 


6.8% 


26.9% 


5.2% 


28.9% 


5.9% 


27.9% 


3.01-4.00 


1 .8% 


1 8.6% 


3.4% 


21.6% 


2.6% 


20.1% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


Mean GPA for first-year science courses 


0.71 


2.12 


0.80 


2.25 


0.76 


2.18 


Pass rate for first-year science courses 


39.8% 


87.8% 


44.4% 


90.2% 


42.2% 


89.0% 
















GPA in first-year Social Studies courses: 














0.00-1.00 


73.6% 


24.0% 


72.2% 


20.9% 


72.8% 


22.4% 


1.01-2.00 


1 5.3% 


27.9% 


1 8.6% 


29.1% 


17.1% 


28.5% 


2.01-3.00 


8.9% 


28.2% 


7.7% 


29.9% 


8.2% 


29.0% 


3.01-4.00 


2.2% 


1 9.9% 


1 .6% 


20.1% 


1 .9% 


20.0% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


Mean GPA for first-year Social Studies 
courses 


0.79 


2.16 


0.84 


2.22 


0.82 


2.19 


Pass rate for first-year Social Studies 
courses 


42.6% 


87.4% 


47.9% 


89.4% 


45.4% 


88.4% 
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It is also quite interesting to note that a large number 
of graduates did rather poorly in core academic subjects 
despite eventually graduating. In math, for example, well 
over half (56%) of eventual graduates had a G PA of 2.0 
or less in their math courses, including 28% with a 1 .0 or 
less (equivalent to a D average). With a two-year combined 
total of nearly 7500 graduates, this is more than 2100 
students in just these two cohorts alone; in other words, 
MPS annually graduates more than 1000 students 
whose mean math GPA is a D average or worse. The 
picture is again only marginally better in the three other 
core subjects, with large numbers of students earning 
diplomas despite very meager records of achievement. 
One clear policy implication from this finding is that simply 
focusing on whether students are "on track” to graduate, 
in terms of attaining enough credits to be promoted 
from year to year, should not be considered a sufticient 
goal for an early warning system. Given the importance 
of GPA as a determinant of other key outcomes after 
high school - most notably, the probability of enrolling in 
college -a focus on college/workforce readiness is an 
equally important, and necessary, component of an early 
warning system. 

Retention Rates 

Moving next to a contrast between the grade 9 retention 



rates of dropouts vs. graduates. Table 3 shows, again not 
surprisingly, that more than half of first-time MPS 9th 
graders in 2001-02 and 2002-03 combined who dropped 
out had been retained either once (33.4%) or more than 
once (an additional 23.1%) as 9th graders, compared to 
just 9.8% and 1 .2% of those who eventually graduated, 
respectively. The mean number of grade 9 retentions 
among eventual dropouts, accordingly, was nearly five 
times higher than the mean number of retentions for 
eventual graduates (0.57 vs. 0.1 1, respectively). Viewing 
the same data a bit differently, however, produces the 
somewhat surprising, and important, finding that some 
44% of MPS students who eventually dropped out had 
not been retained in grade 9 - and thus would have been 
considered “on track” for graduation if the only outcome 
being tracked was credit attainment.'^ 

Absenteeism 

With respect to absenteeism during the first year of 
high school, the data shown in Table 4 paint a somewhat 
more complex picture. At first glance, there appears 
(not surprisingly) to be a strong relationship between 
absenteeism and dropout likelihood. Students who 
eventually dropped out from the 2001-02 and 2002- 
OS cohorts combined had a mean absence rate during 
their first year of high school (36.3%) that was nearly 



TABLE 3: COMPARISON OF GRADE 9 RETENTION RATES BETWEEN DROPOUTS AND GRADUATES 
AMONG FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 





200 


1-02: 


2002-03: 


2-YEAR COMBINED: 


1 


DROPOUTS 

^(N=1218) 


GRADUATES 

(N=3762) 


DROPOUTS 

(N=1330) 


GRADUATES 

(N=3690) 


DROPOUTS 

(N=2548) 


GRADUATES 

(N=7452) 


Retention in Grade 9: 














Retained in grade 9: 














Retained once 


33.2% 


1 0.6% 


33.5% 


8.9% 


33.4% 


9.8% 


Retained more than once 


23.0% 


1 .5% 


23.2% 


0.9% 


23.1% 


1 .2% 


Not retained 


43.7% 


87.9% 


43.3% 


90.2% 


43.5% 


89.0% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


100% 


100% 


Mean # grade 9 retentions 


0.56 


0.12 


0.57 


0.10 


0.57 


0.11 



1 7 In the Chicago Public Schools (CPS) an “on-track" measure is commonly reported to track student progress at the end of the first year of high school. Students are considered to be on-track 
at the end of their first year of high school if they meet two conditions: attaining at least five credits during the first year of high school and not receiving more than one semester F in core 
academic subjects during their first year of high school. 
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TABLE 4: COMPARISON OF FIRST-YEAR ABSENCE RATES BETWEEN DROPOUTS AND GRADUATES 
AMONG FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 







2002-03: 


2-YEAR COMBINED: 


1 


1 DROPOUTS 

L(f^1218) 


GRADUATES 


DROPOUTS 


GRADUATES 


DROPOUTS 


GRADUATES 


1 


(N=3762) 


(N=1330) 


(N=3690) 


(N=2548) 


(N=7452) 


First-Year Absence rate: 














1 0% or less 


1 7.2% 


67.60/0 


1 8.2% 


69 . 40/0 


1 7.7% 


68.5% 


10.1o/o-20®/o 


1 5 . 70/0 


1 8 . 20/0 


1 8.4% 


1 9 .OO /0 


17.1% 


1 8 . 60/0 


20.1%-30®/o 


1 4.5% 


7.0% 


1 5 . 50/0 


6 . 50/0 


1 5 .O 0/0 


6 . 70/0 


30.10/0- 40®/o 


1 2.0% 


3 . 80/0 


12 . 10/0 


2 . 40/0 


1 2 .O 0/0 


3 . 10/0 


40.1%- 50% 


9 . 10/0 


1 . 60/0 


8 . 90/0 


1 . 20/0 


9 .O 0/0 


1 . 40/0 


50.1%-60®/o 


7 . 80/0 


O. 80/0 


7 . 90/0 


O. 8 O /0 


7.8% 


O. 80/0 


60.1%- 70®/o 


6.3% 


O. 50/0 


6 . 30/0 


O. 40/0 


6 . 30/0 


O. 40/0 


70.1% -80% 


7 . 70/0 


O. 20/0 


4 . 70/0 


0.1 0/0 


6 . 10/0 


O. 20/0 


80.1% -90% 


5.8% 


O. 20/0 


5 . 50/0 


0.1 0/0 


5 . 70/0 


O. 20/0 


90.1%- 100% 


4.0% 


0.1 0/0 


2.6% 


0.1 0/0 


3 . 30/0 


0.1 0/0 


Total 


1 00.0% 


1 0O.Qo/o 


1 0O.Oo/o 


1 0O.Qo/o 


1 00% 


1 0O.Oo/o 


Mean overall absence 
rate (9 months)'® 


39 . 80/0 


1 0 . 60/0 


34.6% 


9 . 30/0 


36.3% 


9 . 70/0 



four times higher than that of sfudenfs who wound up 
graduating (9.7%), and very few students (approximately 
6%) with high rates of first-year absence (defined here 
as 30% or higher) wound up graduating. 

At the same time, however, a surprisingly high share of 
eventual dropouts - more than one-third, in fact (34.8%) 
- had comparatively low rates (20% or less) of first-year 
absenteeism. By comparison, the current year-to-date 
absence rate for all MPS 9th graders as of this writing 
in spring 2010 is 77.0%.'® Clearly, missing even 10% of 
total days should be considered a high rate of absence, 
as it is the equivalent of missing one day every fwo weeks 
of school, buf fhe point here is to call attention to the fact 
that while dropouts indeed have higher rates of absence 
than graduates, it is by no means the case that eventual 
dropouts are all characterized by excessive rates of 
absence. Instead, it might be said that good attendance 



during the first year of high school in MPS is a necessary, 
but not sufficient, condition for graduation. 

A visual depiction of the relationship between graduation 
probability and first-year absence rates is shown in Figure 
3; the relationship is mostly linear, as might be expected. 

Several studies involving early warning work have made 
the case that all absences are not created equal, and 
that greater predictive power for negative outcomes such 
as failing to graduate might be provided by focusing on 
unexcused absences rather than total absences. In order 
to know how much distinction exists in MPS with respect 
to these two categories of absence, we reviewed data 
for several years, and found thaf unexcused absences 
represent 60-65®/o of total absences each year.®® While 
it is indeed possible, therefore, to substitute unexcused 
absences for total absences in early warning analyses. 
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1 8 The overall absence rate shown is for the nine months of the traditional school calendar (September-May), in order to present the most accurate picture of monthly attendance by excluding 
months when few students are in session (June - August). 

1 9 Figure drawn from the MPS data warehouse on May 3, 2010. 

20 MPS absence data include four main categories: unexcused (no excuse provided by parent), non-excused (excuse provided by parent, but for a non-valid reason such as caring for siblings), 
excused (valid excuse provided, such as an illness or doctor appointment), and authorized (absence is due to a valid, school-based activity such as a field trip). For purposes of the early warning 
analysis summarized in this report, “unexcused absences” include the unexcused and non-excused categories, while “excused absences” include the excused and authorized categories. 
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FIGURE 3: GRADUATION RATE BY FIRST-YEAR ABSENCE RATE FOR MPS FIRST-TIME 9TH GRADERS IN 
2001-02 AND 2002-03 (COMBINED) 




less -10% -20% -30% -40% -50% -60% -70% -80% -90% -100% 

Freshman Absence Rates 



there is substantial overlap between the two categories, 
and it might further be argued that time out of the 
classroom represents lost learning time regardless of 
cause. It is also true, as described in results from 
regression analyses later, that absence rate is already a 
statistically significant predictor of graduation, and that 
“filtering" absence by excused/unexcused does not add 
to the overall efficacy of the early warning models in 
MPS. 

In addition to separating absence data by type (excused vs. 
unexcused), another aspect of absenteeism as a predictor 
of graduation probability involves its distribution across 
the school year. For example, early warning research in 
Philadelphia (Balfanz & Herzog, 2005) revealed that high 
rates of absence within the first 30 days of the school 
year were particularly good predictors of which students 
wound up dropping out. They surmise that this is because 
the first month represents the crucial time immediately 
following the transition from middle to high school, which 
for most students involves a move from an elementary 
(K-8) or middle (6-8) school to a much larger and less 



personalized learning environment. Accordingly, their 
research recommended that Philadelphia high schools 
monitor 9th graders’ attendance particularly closely 
during the first month of school. 

In MPS, however, this scenario appears to be only 
partially true, as shown in Figure 4. Dropouts do have 
substantially higher rates of absences in each month, 
including September, than do graduates. At the same 
time, monthly absence rates among dropouts are actually 
at their lowest in September, before rising steadily 
through the remaining months of fall semester. Absence 
rates for dropouts then drop slightly in January at the start 
of spring semester, and then continue to rise until the end 
of the school year (May), when they are at their highest 
point of the year (in the 40-50% range - or missing, on 
average, nearly half oi all possible days of attendance). 
As such, targeting attendance rates during September 
makes sense in that absenteeism is substantially higher 
during this month for eventual dropouts than for eventual 
graduates, but this is no more true in September than in 
other months. 
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FIGURE4: MEAN ABSENCE RATES DURING FIRST YEAR OFHIGHSCHOOL,BYMONTH, FOR DROPOUTS 
VS. GRADUATES IN THE COHORTS OF FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 




Behavioral Incidents 

Research on dropout prevention suggests that student 
behavioral data, most often in the form of fracking 
suspensions, is also a worthwhile inclusion in early 
warning systems. Accordingly, our analysis queried 
student discipline data files to investigate the relationship 
between suspensions and probability of graduafion. 
Since complete and reliable MPS disciplinary data are 
not available for the school year 2001-02, Table 5 shows 
results for first-time 9th graders for the school year 
2002-03. Perhaps not surprisingly, eventual dropouts 
were more likely to have been suspended at least once 
than eventual graduates, and were especially prone to 
multiple instances of suspension (37.3% of eventual 
dropouts suspended more than once vs. 1 4.9% of 
eventual graduates). The mean number of suspensions 
incurred during the first year of high school, similarly, 
was notably higher for eventual dropouts (2.08) than 
for eventual graduates (0.70). In terms of the severity 
of the offenses which led to suspension, this can be 
measured reasonably well by the number of suspension 
days assigned, and the data confirm that suspensions 
incurred by eventual dropouts tend to be more severe 
than those incurred by eventual graduates. Just over half 
(54.0%) of suspension incidences incurred by eventual 



dropouts resulted in zero or one days of assigned 
suspension days, compared to 78.8% of those incurred 
by eventual graduates; conversely, the remaining share of 
suspensions (46.0% for eventual dropouts vs. 21 .2% for 
eventual graduates) resulted in two or more suspension 
days assigned. The mean number of suspension days for 
eventual dropouts was also substantially higher than for 
eventual graduates (5.37 vs. 1 .74, respectively). 

Middle school standardized test scores 

Middle school standardized test scores represent 
another indicator that is commonly considered for 
inclusion as a predictor in early warning systems. Table 
6 shows the distribution of both dropouts and graduates 
among first-time MPS 9th graders in 2001-02 and 
2002-03 combined in terms of grade 8 Wisconsin 
Knowledge and Concepts Examination (WKCE) reading 
and math proficiency levels as well as mean scale scores. 
Wisconsin's assessment system, unfortunately, has no 
test for 9th graders, which would be very useful as both a 
predictor of future success (e.g., high school graduation) 
and, perhaps more importantly, as a “missing link” 
between the currently-tested grades of 3-8 and 1 0. It 
would be possible, of course, to compare the 10th grade 
WKCE scores of eventual graduates and dropouts, but 
this probably makes little sense when considering that 
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TABLE 5: COMPARISON OF FIRST-YEAR SUSPENSION INCIDENCES FOR DROPOUTS AND GRADUATES 
AMONG FIRST-TIME MPS 9TH GRADERS IN 2002-03 





2002-03 FIRST-TIME 9^^ GRADERS: 


DROPOUTS (N=1 330) 


GRADUATES (N=3690) 


Suspensions during first year of HS: 






Suspended once 


1 4.7% 


1 2.7% 


Suspended more than once 


37.3% 


1 4.9% 


Never Suspended 


47.5% 


72.4% 


Total 


1 00.0% 


1 00.0% 


Total days of suspension: 






0 days 


47.5% 


72.4% 


1 day 


6.5% 


6.4% 


2-5 days 


1 6.5% 


1 0.6% 


6-20 days 


1 1 .9% 


6.2% 


1 monfh 


1 5.3% 


4.0% 


1 semester 


2.1% 


0.4% 


Entire school year 


0.2% 


0.0% 


Total 


1 00.0% 


1 00.0% 


Mean # of First-Year Suspensions 


2.08 


0.70 


Mean total days of Suspension 


5.37 


1.74 



many students who drop out do so while 9th graders, 
meaning that they aren’t around to take the grade 10 
test. 

Table 6 shows that the mean grade 8 math scale score 
of eventual graduates from the combined cohorts of firsf- 
fime M PS 9th graders in 200 1 -02 and 2002-03 (n=7452) 
was 674.9, which was more than 25 points higher than 
the mean score of dropoufs (649.3; n=2548). The 
discrepancy between dropouts and graduates was nearly 
identical, at 24.5 points, in reading (670.6 for graduafes 
vs. 646.1 for dropoufs).^' In ferms of proficiency levels, 
however, very low rafes of proficiency among MPS 8th 
graders, particularly in math, make this a much less useful 
basis for differentiafing between eventual dropouts and 
graduates. If one was inclined to use proficiency (defined 
as scoring in either the proficient or advanced category) 
as a predictor of dropping out vs. graduating, for example, 
there is a clear difference between dropouts (of whom 



only 1 7.8% were proficienf/advanced as 8th graders) 
and graduates (43.1% proficient -I- advanced). In math, 
however, proficiency rates were extremely low among 
both eventual graduates (10.6%) and eventual dropouts 
(2.3%), such that proficiency is not particularly helpful in 
distinguishing one group from the other. 

Interestingly, the finding that middle school test scores 
are of somewhat limited value as predictors of dropout 
probability is also noted in early warning research in 
Chicago and Philadelphia. In Chicago, for example, it was 
true, on the one hand, that students who had high test 
scores as 8th graders were more likely to be “on track” 
for graduation than were their peers with lower scores. At 
the same time, however, high scores in grade 8 were no 
guarantee of being on track, nor did low grade 8 scores 
doom students to being off track. Among 2003-04 CPS 
first-time 9th graders, for example, fully one-fourfh of 
studenfs who entered high school in the highest quartile 



21 Note that the WKCE scale changed beginning in 2005-06, so grade 8 scale scores described above (from 2000-01 and 2001-02, when first-time 9th graders used in this comparison would 
have been In grade 8) are substantially different from those reported for 8th graders since 2005-06. 
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TABLE 6: GRADE 8 WKCE READING/MATH PROFICIENCY DISTRIBUTION AND MEAN SCALE SCORE 
FOR GRADUATES VS. DROPOUTS FOR FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 
(COMBINED) 




in terms of grade 8 test scores were off-track by the 
end of their freshman year, and conversely, more than 
40% of those in the lowest quartile in terms of grade 8 
fest scores were on frack af fhe end of grade 9. Chicago 
researchers speculate that this apparent contradiction 
is because high schools themselves, and the transition 
from middle to high school, play a substantial role in 
determining whether students are on track or off-track 



for graduation, in a way that is not always obvious from 
looking af students' educational outcomes prior to 
entering high school. 

In other words, a major conclusion emerging from 
Chicago research - which is mirrored in MPS findings - 
is fhat entry into 9th grade represents a critical, “make or 
break” point that can’t necessarily be predicted by even 
the most robust prior (middle school) outcome data. 



22 Mean scale scores in Table 6 reflect the "old” WKCE scale that was used until 2005'06, when a new scale accompanied the new WKCE containing Wisconsin-customized items in 
reading and math. 

23 The “not tested” count reflects students who either graduated from or dropped out of MPS high schools, but for whom no grade 8 WKCE scores were available. This would be because 
either (a) they were enrolled in MPS as 8th graders, but were not administered the WKCE, or (b) they enrolled in MPS after the grade 8 WKCE had been administered, and then 
remained enrolled long enough to either graduate from or drop out of an MPS high school. 
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TABLE 7: COMPARISON OF FIRST-YEAR MOBILITY RATES BETWEEN DROPOUTS AND GRADUATES 
AMONG FIRST-TIME MPS 9TH GRADERS IN 2001-02 AND 2002-03 







1 200 


1-02: 


2002-03: 


2-YEAR COMBINED: 




E 


(ROPOUTS 

(N=1218) 


GRADUATES 

(N=3762) 


DROPOUTS 

(N=1330) 


GRADUATES 

(N=3690) 


DROPOUTS 

(N=2548) 


GRADUATES 

(N=7452) 


Retention in Grade 9: 




























Mobile 


8.7% 


4.4% 


1 3.4% 


4.3% 


1 1 .2% 


4.4% 


Not mobile 


91.3% 


95.6% 


86.6% 


95.7% 


88.8% 


95.6% 




1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


1 00.0% 


Mean # of school 
changes during first 
year 


0.09 


0.04 


0.13 


0.04 


0.1 1 


0.04 



Mobility 

Mobility during the first year of high school represents 
another risk factor for dropping out, although it appears 
that this variable - at least as measured and reported 
here - is somewhat less useful in predicting dropout 
likelihood than might be expected. Mobility as shown 
in Table 7 captures any student who switched schools 
during their first year of high school based on monthly 
attendance records available from the MPS data 
warehouse. As such, any instance of a student being 
enrolled in a school during a particular month that 
was different from his/her school the prior month is 
captured using this method. Among first-time MPS 9th 
graders across the two cohorts combined (2001-02 and 
2002-03), a much higher share of eventual dropouts 



(1 1.2%) than eventual graduates (4.4%) had changed 
schools at least once during their first year of high school, 
and the mean number of first-year school changes was 
substantially higher for dropouts than for graduates (0.1 1 
vs. 0.04). At the same time, however, the relationship 
between mobility and dropout/graduation probability is 
not entirely straightforward. For one thing, nearly 90% 
of students who eventually dropped out did not change 
schools during their first year of high school. Additionally, 
when mobility is included in logistic regression models 
to predict dropout probability (see upcoming discussion), 
mobility is generally oot a statistically significant correlate 
of dropping out. 
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THE ON-TRACK INDICATOR: STRENGTHS 
AND LIMITATIONS 

In its extensive and groundbreaking work with the 
Chicago Public Schools (CPS), the Consortium for 
Chicago School Research (CCSR) at the University of 
Chicago has developed an “on track” indicator for use 
in identifying studenfs af elevated risk of not graduating 
from high school. The on-frack indicator, as shown in 
Table 8, is based on two criteria; the number of credits 
attained by the end of the first year of high school and 
the number of failing grades received in core academic 
subjecfs (reading, math, science, and social studies). 
More specifically, a sfudenf is considered to be on track 
if s/he has attained at least five full credits (equivalent to 
ten half-credits for fraditional semester-based classes) 
and has no more than one failing semester grade across 
the four core academic subjecfs during their first year 
of high school (Allensworth & Easton, 2005). Promotion 
and graduation requirements are also shown for MPS, 
which currently does not report any type of “on frack” 
measure. 



As CPS has both a district policy for grade 9 promotion 
and the CCSR on-track indicator, it is natural to wonder 
how both measures do in terms of predicting on-time 
graduation. Since the CCSR indicator is more “stringent” 
in that it allows fewer instances of failure in core 
academic subjects, it is perhaps not surprising that the 
on-track indicator is a far better predictor of on-fime 
graduation than is the CPS promotion policy. While only 
40% of students who met the grade 9 promotion policy 
wound up graduating on time (e.g., in four years), 81% of 
those who met the on-track requirements graduated on 
time (Allensworth & Easton, 2005). 

Given the demonstrated utility of the on-track indicator 
for predicting on-time graduation as an outcome in CPS, 
a logical task for fhe development of an early warning 
system in MPS was to create this same measure for 
MPS students using transcript data. One comparison of 
results is seen in Eigure 5, which shows the probability 
of graduating within five years (note that this is different 
from “on-time” graduation) cross-tabulated with on- 
track/off-track status. With 22 credits required for 
graduation from MPS, “on track” for MPS is defined, in 



TABLE 8: PROMOTION AND GRADUATION REQUIREMENTS FOR CPS AND MPS 





GRADE 9 
PROMOTION 
(TO GRADE 10) 


FIRST YEAR OF 
HIGH SCHOOL 
“ON-TRACK” 
INDICATOR 


GRADE 10 
PROMOTION 
(TO GRADE 
11) 


GRADE 11 
PROMOTION 
(TO GRADE 
12) 


GRADUATION 


CPS 


5 credits -I- must 
receive passing grade 
in at least 3 (out of 4) 
core subjects during 
both semesters of 
Grade 9 


5 credits -I- no more 
than 1 totai semes- 
ter F across four 
core subjects for 
entire year 


1 1 credits -1- must 
receive passing 
grade in at least 
3 (out of 4) core 
subjects during 
both semesters of 
Grade 10=“' 


1 7 credits 


24 credits®^ 


MPS 


5 credits 


Not in use 


1 0 credits 


1 6 credits^® 


22 credits^"" 



24 20 hours of community service is also required in CPS for promotion from grade 1 0 to grade 1 1 . 

25 40 hours of community service is also required in CPS for graduation. 

26 To be promoted from grade 1 1 to grade 1 2 in MPS, a student must also be in at least his/her seventh semester (or its equivalent) of high school. See httpy'/mpsportal.milwaukee.k12.wi.us/ 
portal/server.pt/gateway/PTARGS_0_457088_491 66_0_0_1 8/7_37.pdf for a full description of MPS graduation policies which are current as of this writing in spring 2010. 

27 The MPS graduation requirement of 22 credits is effective with the graduating class of 201 4-15. Additionally, a student must also have completed one year, and at least 5 credits, at the school 
from which s/he intends to graduate. A small number of MPS high schools require more than 22 credits for graduation or a different combination of courses required under the 22-credit 
requirement 
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FIGURE 5: GRADUATION PROBABILITY BY ON- 
TRACK STATUS FOR END OF FIRST YEAR IN 
GRADE 9, MPS FIRST-TIME 9TH GRADERS IN 
2001-02 AND 2002-03 (COMBINED) 




On-Track Off-Track 



terms of credit attainment, the same as it is for the CPS 
on-track measure (5 credits earned by the end of the tirst 
year of high school). 

The data show that the on-track measure is highly 
predictive of graduation in MPS: fully 93% of first-time 
MPS 9th graders In 2001-02 and 2002-03 (combined) 
who were on-track at the end of their first year of high 
school wound up graduating within five years, including 
88% of whom graduated “on time" in four years). By 



comparison, only 50% of those who were off-track at 
the end of the first year of high school graduated In five 
years, and an even lower share (36%) graduated on time 
in four years. 

Tables 9 and 10 present additional descriptive 
information with respect to the percentage of MPS 
students in various subgroups that are on-track among 
the cohorts of first-time 9th graders in 2001-02 and 
2002-03 combined. Female students of all racial/ethnic 
groups are substantially more likely than males to be on- 
track, while the breakout by racial/ethnic groups shows 
that on-track rates are much higher for Aslan and white 
students than for Hispanic and black students. As might 
be expected, special education and low-income students 
are similarly less likely than their non-special education 
and non-poor peers, respectively, to be on-track. 

As useful as fhe on-track indicator is in terms of predicting 
which students are likely to graduate, however, the utility 
of this construct within the context of building an early 
warning system is constrained by at least two key factors. 
A first is that classifying a studenf as eifher on-track or 
off-track at the end of fhe school year Is simply foo late to 
prevent many students from dropping out. It is also less 



TABLE 9: MPS ON-TRACK RATES BY STUDENTS’ RACE AND GENDER FOR FIRST-TIME 9TH GRADERS 
IN 2001-02 AND 2002-03 (COMBINED) 



RACE/ETHNICITY: 


FEMALE 


MALE 


TOTAL 


African-American 


58.8% 


45.6% 


52.9% 


Aslan 


86.1% 


71.8% 


79.0% 


Hispanic 


64.5% 


55.4% 


59.9% 


White 


77.4% 


70.4% 


74.0% 


Other 


70.8% 


65.3% 


68.0% 


Total 


64.0% 


53.1% 


58.9% 



TABLE 10: MPS ON-TRACK RATES BY STUDENTS’ ECONOMIC STATUS AND SPECIAL EDUCATION 
NEEDS FOR FIRST-TIME 9TH GRADERS IN 2001-02 AND 2002-03 (COMBINED) 



FREE LUNCH: 


SPECIAL EDUCATION 


NON-SPECIAL ED 


TOTAL 


Yes 


47.3% 


58.3% 


55.6% 


No 


52.2% 


67.3% 


64.6% 


Total 


48.8% 


61.8% 


58.9% 
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than ideal in terms of intervening early with students who 
have not dropped out, but have received very low (albeit 
passing) grades that make future college enrollment 
prospects very dim. On both counts, one of the objectives 
of an early warning system is to provide educators with 
actionable information as early as possible during a 
student’s educational careerthat can be used to preclude 
both of these adverse outcomes, and information that is 
not calculated and reported until the end of the academic 
year is certainly a significant limitation in this respect. Far 
more preferable is a measure made available earlier in 
the year, such as the end of the first marking period for 
which all sfudents districfwide receive final grades; in 
MRS, fhis is fhe end of the fall semester, as the process 
for assigning and recording final grades is nof completed 
on a districtwide basis until late January.^® 

A second key limitation of the on-track measure is 
that it essentially takes a very detailed set of student 
outcome Information (based upon course grades and 
credits obtained) and reduces it to a much simpler, 
binary measure (yes/no, on-track/off-track). While this 
approach is certainly useful both in terms of accuracy 
(e.g., in predicfing graduation) and in placing students 
into two readily-understood categories (e.g., on-track 
vs. off-track), it also discards, in effect, very valuable 
information for identifying how far on-track or off-track 
students are during their first year of high school (or, 
analogously, during middle school and fhe second year of 
high school and beyond). After all, a first time 9th grade 
student may have a G PA of 1.0 or less— the equivalent to 
a D average— but by virtue of passing these classes, may 
likely to be counted as “on track.” 

THE “TOTAL QUALITY CREDITS” INDICATOR 

In response to the limitations of the on-track measure, 
the early warning research conducted by VARC in MRS 
began examining ways of showing a more complete set of 
outcome information associated with students’ academic 
experiences during the first year of high school, bofh 
in terms of credit attainment and final grades received. 



The resulf is a measure called Total Quality Credit (TQC) 
attainment, which combines these two measures (credit 
attainment and final grades) info a continuous variable 
that can be calculated both at the end of fall semester 
of students’ first year of high school (grade 9), as well 
as “backward” into middle school and “forward” to the 
second, third, and fourth year of high school. Additionally, 
by making TQC attainment during the first year of high 
school the outcome variable that is predicted by previous 
(e.g., middle school) outcome data under the two-stage 
recursive modeling structure described above, the utility 
of the TQC construct is enhanced even further, as it 
can be used as an early warning “shortcut" to predict 
the most valued high school outcomes (graduation and 
college enrollment) based upon middle school data, 
rather than waiting until these outcomes actually occur 
among high school students up to six years after initial 
9th grade enrollment. 

TQC AND HIGH SCHOOL GRADUATION 

With this background in mind, it is useful to elaborate on 
how well TQC can be used to predict the two outcomes 
of primary interest in the MRS early warning system (high 
school graduation and initial college enrollment). Figure 
6 shows the ffi/e-year graduation rates of firsf-time MRS 
9th graders in 2001-02 and 2002-03 based upon TQC 
attainment during the first year of high school (year t). 
The data show that more than 90% of students with TQC 
attainment of 0.5 and higher (equivalent to a C average 
and higher) graduate within five years, and that in terms 
of predicfing which sfudents are likely to graduate, there 
is only a slight decline from an A average to a C average. 
Graduation probability begins to decline markedly below 
a C average (0.5 TQC), however, and is below 60% at a 
D average (0.25 TQC). 

Figure 7, by contrast, shows on-time graduation probability 
(e.g., in four years) by first-year TQC attainment, and here 
we observe that the relationship is somewhat more linear, 
in that students with a C average (TQC 0.5) are less 
likely to graduate on time (probability of approximately 



28 Some MPS high schools have grading periods prior to the end of fall semester, but the first districtwide data availability for this purpose are the “final marks" for fall semester that are recorded 
in January each year. 
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FIGURE 6: TOTAL QUALITY CREDIT (TQC) ATTAINMENT IN YEAR T (YEAR 1 OF HIGH SCHOOL) BY 
5-YEAR GRADUATION RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 




Ratio of Total Quality Credits Attained 



FIGURE 7: TOTAL QUALITY CREDIT (TQC) ATTAINMENT IN YEAR T (YEAR 1 OF HIGH SCHOOL) BY 
4-YEAR GRADUATION RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 




Ratio of Total Quality Credits Attained 



80%) than those with a B average (0.75 TQC; graduation 
probability approximately 90%). On-time graduation 
probability again declines markedly below a C average, 
such that fewer than half of those with a D average 
or lower graduate in four years. A key policy quesfion 
emerging from Figures 6 and 7 is whefher MRS wishes 
fo have on-fime graduation or five-year graduation as 
the outcome variable being predicted, or perhaps both. 



There may be some benefit fo having on-time graduation 
as the predicted outcome, given current federal and 
state policies emphasizing on-time graduation, although 
MRS may also be interested in predicting graduation 
probability for students who fake five years given high 
rates of grade 9 retention in the district. 
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PART 3. CREATING THE “TOTAL QUALITY CREDITS” 
INDICATOR 



FIGURE 8: TOTAL QUALITY CREDIT (TQC) RATIO ACROSS HIGH SCHOOL BY 5-YEAR GRADUATION 
RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 (COMBINED) 




Ratio of Total Quality Credits Attained 



FIGURE 9: TOTAL QUALITY CREDIT (TQC) RATIO ACROSS HIGH SCHOOL BY 4-YEAR GRADUATION 
RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 (COMBINED) 




Ratio of Total Quality Credits Attained 



Figures 8 and 9 show how TQC attainment over 
students' entire high school experiences (as opposed to 
the first year of high school, as portrayed in Figures 6 and 
7) relates to graduation probability on both a five-year 
basis (Figure 8) and an on-time basis (Figure 9). Data 
are again for first-time MRS 9th graders in 2001-02 and 
2002-03, although these two cohorts are combined here. 
These two figures show even more clearly the drop-off in 
both types of graduation probability that occur below a C 
average (TQC 0.5). 



TQC AND COLLEGE ENROLLMENT 

The relationship between TQC attainment and initial 
college enrollment, the second outcome of interest 
within the MRS early warning system, is also intriguing. 
College enrollment data for MRS graduates comes from 
the National Student Clearinghouse (NSC), which tracks 
and reports postsecondary enrollment information for 
more than 90% of high school graduates nationwide 
who enroll at the postsecondary level.^® Initial college 



29 MPS began receiving data from NSC in 2008, and has produced an initial report on findings available at http://mpsportal.milwaukee.k1 2.wi.us/portal/server.pt/gateway/ 
PTARGS_0_2_52905_0_0_18/PSReport_03252009.pdf. 
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FIGURE 10: TOTAL QUALITY CREDIT aOC) ATTAINMENT DURING YEAR T (YEAR 1 OF HIGH SCHOOL) BY 
INITIAL COLLEGE ENROLLMENT RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 




Ratio of Total Quality Credits Attained 



FIGURE 1 1 : TOTAL QUALITY CREDIT (TOC) ATTAINMENT ACROSS HIGH SCHOOL BY INITIAL COLLEGE 
ENROLLMENT RATE FOR FIRST-TIME MRS 9TH GRADERS IN 2001-02 AND 2002-03 (COMBINED) 




enrollment, as opposed to the more meaningful 
measures of college persistence and graduation, is 
used as the outcome variable for the time being, as this 
is the only outcome that can really be predicted using the 
NSC data obtained by MRS to date. MRS has received 
college enrollment data for its graduating classes of 
2005-09; while a small number of these students (less 
than 100) have obtained two-year (associates) degrees, 
very few have been enrolled long enough to obtain four- 
year degrees. Clearly, college graduation is the long- 



term variable of the greatest interest given its impact on 
lifetime earnings, and the MRS early warning system will 
switch to using this as the second outcome variable to 
be predicted (joining high school graduation) as soon as 
sufficient data exist for this purpose. 

Figure 10 shows initial college enrollment rates by TQC 
attainment during the first year of high school (year f), 
while Figure 1 1 shows inifial college enrollmenf by tofal 
TQC attainment across four years of high school (years 
t fhrough f-l-3). A generally linear pattern emerges, in 
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PART 3. CREATING THE “TOTAL QUALITY CREDITS” 
INDICATOR 



which probability of initial college enrollment steadily 
increases with TQC attainment level; in other words, 
students whose TQC attainment is the equivalent of an 
A average, both in year t and across four years of high 
school, are observably more likely to enroll in college than 
those with the equivalent of a B average, who are in turn 
more likely to enroll than those with a C average. Very 
few sfudents who graduafe wifh a D average or below, by 
contrast, enroll in college. Both figures show, for example, 
thaf approximately 80% of sfudenfs with first-year TQC 
attainment of 1.0 (equivalent to all A's in core-subject 



courses) enroll in college, compared to about 60% 
of those with TQC of 0.75 (equivalent to a B average 
in core-subject courses) and about 40% of fhose wifh 
TQC of 0.5 (equivalent to a C average). At a TQC level of 
0.25, equivalenf to a D average, college enrollment rates 
decline to less than 20%. This finding stands in partial 
contrast to TQC attainment and graduation probability 
shown previously, in which students with the equivalent 
of a C average were only marginally less likely than fheir 
peers with an A average to graduafe, but the probability 
of graduation fell off markedly below a C average. 



PART4.COMBININGRISK FACTO RS I NTO AN EAR LY 
WARNING SYSTEM 



Having investigafed the descriptive relationships 
thaf exisf between individual independent (predictor) 
variables (such as academic attainment as measured by 
TQC, attendance, mobility, retention, etc.) and the desired 
high school outcomes (graduation and initial college 
enrollment), the next step was to combine these risk 
factors into predictive models for understanding TQC 
attainment at the end of the first year of high school. In 
the broadest sense, this work seeks to uncover which 
combinations of risk factors, at various grade levels 
following entry into grade 6 and continuing through 
high school, provide the greatest predictive power for 
understanding which MRS students are at the highest 
risk ot not graduating and not enrolling in college.^” 

To investigate how well various combinations of middle 
school outcomes (years t-1 , t-2, and t-3) predict first-year 
of high school TQC, and to determine which middle school 
predictors serve as the most important predictors of first- 
year TQC, a series of regression analyses using first-year 
TQC attainment among first-time MRS 9th graders in 
2005-06 and 2006-07 (combined) as the dependent 
variable were prepared. Independent (predictor) variables 
used in the analyses include the following: 



School type from which a student graduated” from 8th 
grade (K-8 vs. traditional middle school vs. middle/high 
school) 

• Grade 8 WKCE reading and math proficiency level 

• Absence rates during each year of the middle 
grades 

• TQC attainment during each year of the middle 
grades 

• Instances of mobility during each year of the middle 
grades (mobile vs. not mobile) 

• Retention during each year of the middle grades 
(retained vs. not retained) 

• Suspensions during each year of the middle grades 
(suspended vs. not suspended) 

• Student demographics (gender, race/ethnicity, ELL, 
special education, economic status) 

A summarized version of results is shown in Table 1 1, 
and inferential findings are discussed in the next section. 



30 It is worth reiterating here, as discussed above, that under the two-stage recursive data modeling structure used in this project first-year TQC attainment is the outcome variable being predicted, 
rather than the end-of-high school outcomes themselves. This is because first-year TQC serves as a “shortcut" to the two desired long-term early warning outcomes (graduation and college 
enrollment) by being highly predictive of these two outcomes and because first-year TQC can itself be predicted using middle school outcomes. It is also worth emphasizing that the results 
described are preliminary, in the sense that much more work is anticipated in order to develop the most useful and accurate early warning tools for MPS. 
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Thus far, three main inferential findings have emerged 
from our analyses, although given the interim nature of 
this report, we expect that additional findings of interest 
will emerge In the near future. These Include the following; 

Finding #1. “High yieid” indicators of students 
at risk for dropping out of MPS and graduating 
with iow coiiege/workforce readiness inciude 
academic achievement (TQC), attendance, 
suspension, and retention/overage. 

Looking at the results of the regression analyses in 
Table 1 1, it was possible to identify a set of “high yield” 
indicators-- those independent variables that showed 
a statistically significant effect on first-year TQC— for 
purposes of identifying students at risk. In the case of 
the MPS early warning system, academic achievement 
( as measured by TQC), attendance, suspension, and 
retention/overage as the four emerged as the most 
important predictors. 

While Grade 8 WKCE performance in both reading and 
math also showed a significant association with first-year 
TQC (e.g., students who are proficient and advanced in 
8th grade usually have higher first-year TQC, and vice 
versa), it is worth recalling from Table 6 that 8th grade 
WKCE performance does not provide a consistently 
useful basis for differentiating dropouts from graduates, 
due in large part to the fact that so many MPS students 
who did eventually graduate performed poorly (in terms 
of both scale score and proficiency level) on the test.^' 
And while Table 1 1 shows that school type (K-8 vs. 
traditional middle school) had a statistically significant 
effect on the dependent variable, given the limitations 
to the construct validity of this variable we feel it 
warrants further investigation before being identified as 
a “high yield indicator.” These results and limitations are 
addressed in a separate discussion below. 

It is interesting to note that among the demographic 
variables included in the model, special education status 
has a positive coefficient, which is perhaps different 
than what might expected given that special education 



students tend to have lower rates of graduation and 
college enrollment than their peers. This appears to 
be a product of the decision to include all core-subject 
courses in TQC calculations, rather than just those core 
courses that count toward MPS graduation requirements. 
In other words, by incorporating all core courses in 
TQC calculations - including the remedial, or “helper," 
courses noted previously, which many special education 
students take, and presumably fare better in than they 
do in “regular” core courses - special education status 
appears to be positively associated with TQC attainment, 
when we strongly suspect that this is not really the 
case. Subsequent analyses have applied a “partial” 
(0.5) weighting to remedial courses in order to strike an 
appropriate balance between excluding these outcomes 
completely and including them at a “full” weighting. 

Finding #2. The ability to predict first year high 
school outcomes thus far using middle school 
indicators is somewhat limited. Among other 
implications, this suggests that the transition to 
high school presents a significant challenge for 
many middle grades students in MPS, even those 
with relatively high levels of performance through 
the middle grades. 

Qne very important observation is that using even the 
most complete sets of middle school outcomes provides 
somewhat limited predictive power in terms of predicting 
end of first-year high school (year t) TQC attainment - 
and having multiple years of middle grades outcomes 
appears to be minimally, if at all, helpful in predicting first- 
year TQC attainment. The r-squared values for predicting 
first-year TQC using all combinations of middle school 
predictors (t-1 or 8th grade alone, t-1 -I- 1-2 together, etc.) 
are all in the 0.25 - 0.35 range, meaning that only 25- 
35% of the variance observed in the dependent variable 
(first-year TQC) is explained by the sets of independent 
(predictor) variables shown in Table 1 1 . The r-squared 
value from using just t-1 (grade 8) variables to predict 
first-year TQC, for example, is 0.36, with no additional 
predictive power generated by adding t-2 (grade 7 for 



3 1 It might be useful to incorporate WKCE data into regression analyses as a scale score (e.g., a continuous variable) either in place of or in additional to its use as a proficiency level (e.g., a 
categorical variable). Since the WKCE scale changed from the 2004-05 to the 2005-06 test, however, using proficiency level provides a more consistent Interpretation of results across time 
given that two separate cohorts (first-time MPS 9th graders in 2005-06 and 2006-07) are used in the analysis. 
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ABLES ALONE T-2 VARIABLES T-3 VARIABLES ABLES ALONE VARIABLES ABLES ALONE 



TABLE 1 1 : REGRESSION MODELS FOR PREDICTION OF FIRST-YEAR TOC ATTAINMENT USING 
MIDDLE GRADES (T-1, T-2, T-3) OUTCOMES 
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most students) data and only very minimal power from 
adding t-3 (grade 6 for most students) data. R-squared 
values drop even more when t-1 (grade 8) data are 
excluded from the models, dashing the notion that we 
know as much about predicting first-year TQC in years 
t-2 and t-3 as we do in year t-1 . 

In other words, what appears to be happening is that 
many MRS students who have done reasonably well 
throughout middle school in terms of both academics and 
engagement (e.g., attendance and behavior) experience 
substantial academic deficiencies during their first year of 
high school. As shown previously in descriptive findings 
on MRS dropouts, these negative experiences during 
the first year of high school often culminate in students 
either repeating 9th grade or dropping out altogether - 
whether during the first year of high school itself or over 
a longer process of disengagement that may persist into 
the second, third, and even fourth years. Additionally, it 
does not appear that first-year TQC can be predicted 
as well using 6th and 7th grade outcome data as can 
be done using 8fh grade outcomes, so adding predictor 
variables from additional years prior to grade 8 does not 
help much. 

Finding #3. Irrespective of school type (K-8 vs. 
traditional middle), some MPS middle schools 
appear to be doing a better job than others in 
preparing students for the rigor of high school. 

The inclusion of school-level indicators to identify both the 
middle school from which each MRS student "graduated” 
as well as the school(s) they attended during their first 
year of high school also told an interesting story. Looking 
at the relationships between middle school outcomes and 
first-year TQC, both for the district overall and broken out 
by school type (e.g., K-8 vs. traditional middle school), 
obscures important differences that may exist between 
individual schools which MRS students attend in the 
middle grades. In other words, it may be the case that 
stronger relationships between middle grades outcomes 
and first-year TQC emerge for some schools than for 
others. If so, this would suggest that some schools are 
doing a better job than others in preparing students for 



the rigor of high school. It may further suggest that having 
the right "match” of graduating 8th grade student to high 
school acts as a critical determinant of success at the 
high school level and beyond, in much the same way that 
the matching process of high school graduate to college 
has been shown to be very important as a determinant of 
college success (Roderick et al, 2008). 

Before delving into results which investigate this 
hypothesis, there are a number of issues inherent in 
studying school-level predictive relationships between 
outcomes at the middle grades and those at the high 
school level that should be mentioned briefly. A first is 
that the school from which a student "graduates” from 
middle school is represented somewhat incompletely 
in this analysis, as what is used here is only the school 
from which a student completed 8th grade rather than 
a complete accounting of mobility in grades 6-8. With 
respect to the notion of "matches” between the middle 
and high schools that students attend, furthermore, MRS 
has no clearly-defined feeder patterns through which 
students from a selected set of middle or K-8 schools are 
"tunneled” to specific high schools. This means that there 
are a great many potential matches between "sending” 
middle/K-8 schools and “receiving” high schools - with 
small numbers of students involved in many of these 
matches - making it difficult to track this process in a 
meaningful way. Additionally, the high school landscape 
in MRS has evolved substantially over the past 5-10 
years, with considerable numbers of schools opening and 
closing, converting to charter status or reverting back to 
"traditional” status, adding or dropping grades, etc., which 
further complicates an already-complex situation. 

With these caveats in mind, a sampling of results 
comparing the relationship between middle grades 
outcomes and first-year TQC for individual MRS schools 
that enroll middle grades students appears in Table 1 2. 
Results are restricted to 33 schools which had at least 
50 “graduating” 8th graders in the 2005-06 and 2006- 
07 school years combined, and schools are assigned 
a fictitious school code at this point due to the very 
preliminary nature of the results. 
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TABLE 12: REGRESSION RESULTS FOR PREDICTION OF FIRST-YEAR TOC ATTAINMENT USING 
MIDDLE GRADES (T-1, T-2, T-3) OUTCOMES, BY “GRADUATING” 8TH GRADE SCHOOL SITE 
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0.4291 


13 


81 


0.2138 


0.1 179 


0.4284 


14 


427 


0.4854 


0.0452 


0.4277 


15 


400 


0.4303 


0.0520 


0.4229 


16 


455 


0.3831 


0.0538 


0.4028 


17 


486 


0.5421 


0.0613 


0.3946 


18 


430 


0.3668 


0.0493 


0.3918 


19 


63 


0.5079 


0.1754 


0.3886 


20 


94 


0.5923 


0.1485 


0.3831 


21 


330 


0.3415 


0.0502 


0.3818 


22 


252 


0.3678 


0.0674 


0.3815 


23 


405 


0.3830 


0.0498 


0.3807 


24 


80 


0.2166 


0.2023 


0.3773 


25 


394 


0.1953 


0.0412 


0.3733 


26 


99 


0.4761 


0.1260 


0.3698 


27 


554 


0.3140 


0.0428 


0.3568 


28 


420 


0.4763 


0.0551 


0.3559 


29 


91 


0.4614 


0.1625 


0.3439 


30 


337 


0.2932 


0.0673 


0.2984 


31 


77 


0.3089 


0.1380 


0.2956 


32 


117 


0.3146 


0.1978 


0.2931 


33 


53 


0.5784 


0.3054 


0.2028 
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Looking at these results, there does appear to be 
support for the notion that some schools are doing a 
better job than others in preparing students for high 
school. The r-squared values for individual middle/K-8 
schools vary subsfantially, from instances in which the 
predictive relationship between middle grades outcomes 
and the TQC attainment of these same students at the 
end of their first year of high school is fairly high (with 
r-squared values of 0.64, etc.) to other instances where 
the predictive relationship is much lower (with r-squared 
values in the 0.2 range). 

Of course, more in-depth exploration and confirmation is 
clearly needed. For example, it would be very useful to 
investigate whether urban districts other than MRS have 
uncovered more robust predictive relationships between 
outcomes in the middle grades and those early on in 
students’ high school careers, and whether this notion 
of “pairings” between middle schools and high schools 
appears to be important. 

IMPACT OF SCHOOL TYPE 

Investigating the impact of school type upon prediction 
of first-year TQC is based upon prior research (Neild, 
Stoner-Eby, & Furstenberg, 2008; Allensworth & Easton, 
2005; Balfanz & Legters, 2006) which has identified 
the transition from the middle grades to high school 
as a critical point in the educational trajectory of many 
students. This research raises the question of whetherthe 
relationship between first-year TQC and middle grades 
outcomes for the district overall might be obscured by 
the type of school attended by MRS students during the 
middle grades. 

Beginning in the early 2000s, MRS (as well as other urban 
districts) converted many of its traditional elementary 
schools (e.g., those with grade configurations of K-5 or 
K-6)toaK-8 configuration, based in large part on research 
suggesting that outcomes in middle grades (including 
both academic achievement and measures of sfudenf 
engagement such as attendance and behavior) would 



improve by eliminating the transition from elementary to 
middle school and by providing a more nurturing learning 
environment (see Juvonen et al, 2004). As a result, the 
number of traditional (6-8) middle schools in MRS has 
dwindled from 31 in 1998-99 fo 1 1 as of fhe 2009- 
1 0 school year,^^ compared to 63 schools with a K-8 
or 1-8 configuration in 2009-10. The reduced number 
of remaining middle schools in 2009-10, however, are 
large enough fhat more than one-third (34.2%) of the 
district's total enrollment of 15,656 in grades 6-8 is 
found in tradifional middle schools, compared to 56.4% 
of 6-8 enrollment in K-8 sites and 9.0% in elementary/ 
secondary schools with grade configurations such as 
6-1 2 or 7-12. 

To investigate whether one type of middle grade 
configuration (K-8 vs. traditional middle school) produced 
superior outcomes for students, MRS commissioned two 
evaluations on the topic (see Cook, 2005; Cook & Bloom, 
2007). The first concluded that on selected measures of 
student engagement, academic achievement/attainment, 
and value-added growth measures, K-8 schools seemed 
to outperform traditional middle schools. The second 
evaluation again found superior results for K-8 schools 
in the areas of student engagement and achievement/ 
attainment, but also found that value-added growth was 
significantly higher in traditional middle schools. 

It is important to emphasize that these evaluations 
focused on differences in student outcomes during the 
middle grades themselves, rather than outcomes at the 
high school level. In other words, these evaluations did 
not specifically address the question of whether either 
school type produces more favorable results after 
students transition to high school. 

Returning to the M RS early warning discussion, a question 
of obvious interest given MRS's move toward K-8 schools 
is whether school configuration type during the middle 
grades (K-8 vs. traditional middle school) influences the 
ability to predict TQC attainment during the first year of 
high school. Accordingly, we included school type as a 



32 In addition to the 1 1 schools with a 6-8 grade configuration in 2009-10, MPS has two schools with a 6-9 grade configuration that are classified as middle schools by the Wisconsin Department 
of Public Instruction. 

33 A very small number of MPS middle grades students (0.3%) were enrolled in high schools In 2009-10. 
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variable in regression equations, based upon the school 
in which students completed 8th grade. One hypothesis 
here was that K-8 schools might be less effective than 
traditional middle schools in terms of preparing students 
for the academic rigor and social adjustment of high 
school; this might be because middle schools by design 
have several key structural components (including 
elective classes, a daily bell schedule, and mostly different 
teachers for each subject) that more closely resemble 
high school than is the case with the more “elementary- 
centric” model of the K-8 school. It might further be 
hypothesized that traditional middle schools use more 
rigorous and demanding grading practices than do K-8 
schools, perhaps because middle schools were originally 
conceived some 50 years ago as a “halfway” point, or 
transition, from elementary to high school. 

Regression results shown in Table 1 1 provide some 
support for the notion that the relationship between 
middle grades outcomes and first-year TQC is stronger for 
students who complete 8th grade in a traditional middle 
school. In the regression itself, if is necessary to choose 
one of the three main school types that MRS students 
attend in the middle grades (K-8, middle, or middle/high) 
as the reference group against which an effect of school 
type can be compared. Traditional middle schools were 
chosen for this purpose, meaning that the coefficients 
for K-8 and middle/high schools reflect either a positive 
or negative effect on the dependent variable (first-year 
TQC) relative to middle schools. The negative (and 
statistically significant) effect observed for K-8 schools 
indicates that students who “graduate” from 8th grade in 
a K-8 school are at a disadvantage relative to their peers 
who complete grade 8 in a traditional middle school after 
holding constant all other predictors. 



This finding should be interpreted with caution, as using 
school type as an independent variable in this manner (e.g., 
capturing the school from which a student “graduates" 
from 8th grade) does not necessarily account for the 
length of time a student spent in a particular school type 
during the middle grades. In other words, just because a 
student completed 8th grade in a K-8 school does not 
mean that s/he spent all, or even most, of his/her middle 
grade enrollment period in that same type of school; s/he 
could have transferred to a K-8 school from a traditional 
middle school sometime during grade 8, or even in grade 
6 or 7. As such, a more complete accounting for the 
effect of school type during the middle grades would 
perhaps limit analyses to include students who started 
and completed grades 6-8 in the same school type (K-8 
vs. middle), although high rates of mobility during the 
middle grades would likely limit sample size considerably. 



The Senior Urban Education Research Fellowship Series, Volume II - Fall 2010 



49 



FINDINGS 





FINDINGS 



FINDINGS/DISCUSSION 



OTHER VARIABLES OF POTENTIAL 
INTEREST 

Several other completely different explanations may 
also exist for the findings encountered thus far, and 
possibilities such as these form the next phase of early 
warning work in MPS. It may be the case, for example, 
that the information currently available is judged 
insufficient for producing predictions for the outcomes of 
interest (graduation and college enrollment/graduation), 
and that additional information from the middle and/or 
high school grades is needed. One potential source of 
additional information might be some type of academic 
“stress test” administered at one or more points during 
the middle grades that was somehow able to tap into 
skills and constructs needed for success in high school 
that are not currently addressed by existing measures 
such as final course grades. This might be an academic 
exercise that is either already in use in the district or could 
be incorporated fairly easily (for example, a research or 
term paper assigned to 7th graders each year); another 
possibility might be a “skills inventory" for each student 
filled out by their teachers each year in the middle grades 
that measures teacher perceptions of student readiness 
for success at the high school and college levels. 



Developing a way to measure students' level of 
engagement with, and “connectedness” to, their middle 
and high schools may also offer a useful source of 
information for predicting which students are less likely 
to graduate and enroll in/graduate from college. Several 
surveys which purport to measure various combinations 
of these psychological constructs are known to exist, 
including Gallup Student Poll, the EdVisions HOPE survey, 
and the High School Survey of Student Engagement. 
Research (see, for example. Close & Solberg, 2008) has 
indicated that tapping into these constructs can be very 
useful for predicting certain outcomes, such as higher 
attendance rates and improved GPA. MPS currently 
administers an annual Climate Survey to all students in 
grades 4 and higher that might incorporate these kinds 
of measures, but at the present time two substantial 
constraints exist: first, the current Climate Survey 
contains limited information that specifically addresses 
students’ perceptions of being connected to their school; 
and second, no student-level response data are collected, 
so it would be impossible to connect results to academic 
outcomes within an early warning system in a way that 
identified students at elevated risk. As such, it might 
be useful to investigate the feasibility of augmenting 
the district's existing Climate Survey with more robust 
measures of student “connectedness” to their school. 
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USING EARLY WARNING RESULTS TO 
IMPACT PRACTICE 

While the research component of the MPS early warning 
system conducted by VARC continues, we are pleased 
to report that the district has already taken steps to 
incorporate early warning findings into “products” 
intended to inform district improvement efforts. MPS 
has developed an “early warning dashboard” at the 
high school grades, for example, that is known as 
SAIL (Student Academic Indicators for Learning). SAIL 
provides up-fo-dafe informafion for the four “high yield" 
indicators identified in our work and in prior research 
on early warning and dropout prevention efforts (e.g., in 
urban districts such as Chicago and Philadelphia): Total 
Quality Credits, attendance, suspensions, and retention/ 
overage. Each of the four metrics has three distinct risk 
groups—green (low risk), yellow (moderate risk), and red 
(high risk)— based upon thresholds identified by early 
warning research. 



A “screenshot” of this school-level summary page is 
shown below in Figure 12. A student-level detail report 
(not shown) is also accessible from this page, offering 
“drillable,” student-level data, such as a list of students 
whose status on that particular indicator has placed 
them into a specific risk category. Data used to populate 
these two reports are refreshed as frequently as possible, 
ranging from annually (for the retention/ overage metric) 
to semi-annually (TQC attainment, based upon final 
marks awarded at the end of each semester) to daily (in 
the case of atfendance and suspension data obtained 
from the MPS data warehouse). This data also “moves” 
with each student as s/he changes schools, a very helpful 
feature given high rates of mobility among MPS students. 



FIGURE 12: SCREENSHOT OF SCHOOL-LEVEL SUMMARY EARLY WARNING PROTOTYPE REPORT 




Total Quafity Credits Overall 




52 



The Council of the Great City Schools 












Our next step will be to investigate the feasibility of 
providing further detail within each of the risk categories 
shown in Figure 12 by developing and assigning to 
each student a “Risk Index Score” that provides a 
more specific estimate of risk level at various points in 
time for the two key outcomes identified in our work: 
dropping out and graduating with low levels of college 
readiness. The starting point for these student-level 
probability estimates will be the end of 8fh grade; in 
other words, based upon all available information as of 
the end of 8th grade, we will generate specific probability 
estimates for each student expressing their probability 
of both outcomes as of that point in time. As additional 
information becomes available following each student's 
entry into high school, the probability estimates will be 
updated on a regular basis; while it would be possible to 
do this literally every day as attendance and suspension 
data are refreshed nightly, it seems more practical to 
start with perhaps monthly updates. Working “backward” 
into the middle grades, analogously, our plan is to create 
similar probability estimates at predetermined times (e.g., 
halfway through and at the end of each grade in middle 
school). As additional data on college persistence and 
graduation become available in the future, it may also be 
possible to include these as third and fourth outcomes to 
be predicted. 

POLICY IMPLICATIONS 

Several broader policy implications and avenues for 
future work both within and outside of the district have 
also emerged from VARC's early warning work in MRS. 
This work has contributed to a deeper understanding 
of the dropout phenomenon in MRS, which has been 
a persistent issue in the district and many of its urban 
counterparts for many years. By simply knowing more 
about dropouts in a descriptive sense, for example - such 
as when they drop out most frequently during their high 
school career - it is possible to design interventions to 
more effectively address this problem. 

For example, the fact that many MRS dropouts persist 
well into their high school careers suggests that 
dropping out is best understood as a gradual process of 



disengagement rather than a single catastrophic event, 
and that efforts to better identify at-risk students based 
on observable outcomes during the middle grades and 
even the first year of high school could prove critical. At 
the same time, the fact that the modal grade for MRS 
dropouts is grade 9 suggests that additional programming 
targeted at easing the difficult transition from middle to 
high school is warranted, and that this programming 
should be informed by rigorous early warning data. 

While early warning work conducted to date has been 
very promising, another policy implication emerging from 
our work is that additional research and development 
is clearly needed to both “fine tune” and expand these 
systems. For example, our analysis illustrates that it is 
misleading to think of graduates as a single, monolithic 
group whose trajectories for life success are largely similar. 
Instead, we identify a second key outcome that should 
be predicted by early warning systems, in addition to 
predicting which students are at elevated risk of dropping 
out: those students who are likely to graduate, but with 
very low levels of college and workforce readiness. These 
are students who have done just enough to continue 
accumulating credits - often by receiving D grades - 
but for whom the prospects of a meaningful career, with 
or without some type of postsecondary education - are 
extremely limited. The data show that very few of these 
students go on to college, and the fact that college and 
workforce readiness are increasingly thought of as very 
similar skill sets does not bode well for students entering 
the workforce after a lackluster high school career. 

Moreover, the relatively low predictive power of even 
the most complete sets of middle grades outcomes 
suggests that additional data are needed in order to 
make early warning models more robust. One area that 
warrants further investigation, for example, is the notion 
of “matches” between individual middle schools and high 
schools. Future early warning work may also benefit 
from the development of measures of key non-cognitive 
outcomes such as students’ sense of engagement in 
school and the efficacy of their support networks, as well 
as non-school predictive information such as students' 
health and family backgrounds. 
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APPENDIX 



DEVELOPING NEW HIGH SCHOOL VALUE-ADDED 
INDICATORS IN MPS 



The second major line of work supported by the Senior 
Urban Education Research Fellowship, in addition to the 
Early Warning system, has involved the development 
of new value-added performance indicators for MPS 
high schools. As noted in the introductory section of 
this report, one of the ways in which the Fellowship has 
been particularly useful to the MPS/WCER research 
partnership is that it has contributed significantly to the 
expansion of our work to the high school level. Within 
the past year, VARC has completed a comprehensive 
evaluation of MPS high schools and produced the first 
longitudinal analysis of initial college enrollment rates 
among MPS graduates from the classes of 2005-09. 
Additionally, MPS has issued a contract to VARC to 
evaluate several new initiatives at the high school level, 
including a “Freshman Connection" program designed to 
produce smoother transitions from the middle grades to 
high school and a credit recovery program at year-round 
high schools. 

VARC has produced annual estimates of value-added 
performance for all MPS elementary and middle schools 
since 2001 , and has recently (as of this writing in spring 
2010) released the most recent version of these data 
incorporating 2009-10 test results. Due to the fact that 
Wisconsin's state testing program includes only grades 
3-8 and 1 0, however, it has not been possible to produce 
value-added data for MPS high schools for the past 
several years, as there has been no test in grade 9 with 
which to measure growth to grade 1 0. The district did 
administer TerraNova assessments to 9th grade students 
prior to the start of statewide 3-8 and 1 0 testing in 
2005-06, and VARC was able to produce several years 
of value-added estimates for grade 9-10 growth using 
these data. No value-added data have been produced for 
MPS high schools for the past several years, however, 
following the discontinuation of grade 9 TerraNova 
testing in 2005-06. 

In response to both the value-added “void” that has existed 
at the high school level in MPS as well as the district's 
deepening commitment to using value-added indicators 
as measures of school performance and accountability, 
VARC has used the Senior Urban Education Research 



Fellowship as a catalyst for developing pilot versions 
of several new value-added performance indicators for 
MPS high schools. Two specific pilot indicators of high 
school value-added developed under the auspices of the 
Fellowship work include the following: 

• Within-year (September-May) value-added growth 
on MPS grade 9 quarterly benchmark exams; 

• Attendance value-added for first-time MPS 9th 
graders using prior attendance (from grade 8) as a 
control; 

Addressing first the pilot value-added system based upon 
grade 9 quarterly benchmark testing data, VARC's work 
in building value-added models for school improvement 
and accountability based upon state standardized tests 
such as the WKCE has had at least two key limitations 
for which there is no true solution under the existing 
testing system: 

• There is a substantial “lag time” between when 
students take the WKCE (November) and when 
districts receive results (March/April), and this 
delay is exacerbated even further in the calculation 
of value-added dafa by the fact that value-added 
measures growth from year to year. As such, results 
from Year 2 of the value-added model are based 
upon tests that a student took some 1 8 months 
earlier; 

• The mid-year (November) testing window associ- 
ated with the WKCE (and some other state tests, to 
a lesser degree) greatly complicates value-added 
calculations and interpretations/uses, as growth 
must be divided across two school years (and, by 
association, two grade levels and classrooms, and 
sometimes two completely different schools). 

In response to these limitations, there has been 
substantial growth in recent years in the number of 
districts and schools using shorter-cycle standardized 
tests known generally as “benchmark” assessments. 
These tests, typically administered 3-4 times annually to 
students in grade ranges from the early elementary to 
early high school grades, are ideal for expanding value- 
added beyond yearly NCLB-type assessments in that 
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they (a) return results to districts much more quickly for 
use in school improvement, and (b) address the issue 
of mid-year testing by allowing growth to be compared 
across more intuitive starting and ending points than is 
currently the case with the November-November period 
defined by the WKCE testing cycle. MRS launched its 
own benchmark testing system for students in grades 
3-9 in reading and math during the 2006-07 school 
year. These tests, designed by Discovery Education, have 
been administered in September, December/January, 
March, and May of each year, thus allowing growth to 
be measured across a variety of different intervals 
(September-May, September-January, etc.) which do 
not require growth to be apportioned across multiple 
school years. The utility of benchmark testing for MRS 
schools is evident in the fact that while participation 
has been optional (other than for 9th graders, for whom 
participation is mandatory per MRS board policy), the vast 
majority of schools have voluntarily chosen to participate 
in the program. 

At its core, the VARC value-added model is a regression 
of current achievement on past achievement, a set 
of student characteristics, and a full set of school or 
classroom indicators. A school-level model can be 
described using the following regression model: 

yi, = f + Xyoi + pXi + (XSi + e, 

where y,, is student /’s current achievement, is 
student i’s past achievement, X. is a vector of student 
i’s characteristics, S^. is a vector of variables indicating 
the schools student /' attended, and e^. is the component 
of current achievement that cannot be explained with 
past achievement, student characteristics, or schools. 
The coefficients on the school indicators a is a vector of 
parameters that become the schools' value-added. 

This model uses data on past test scores and student 
characteristics to produce a prediction (more specifically, 
a best linear prediction) of current test scores. A school's 
or classroom's value added is equal to the number of 
extra points on the test that students in that school or 



classroom gained on the test relative to that prediction. 
Eor example, a school in Wisconsin may have a value- 
added of -1-3 WKCE points; this means that students at 
that school gained 3 more points on the WKCE from one 
year to the next than did observationally similar students 
across the district. 

Since test scores used in value-added modeling do not 
perfectly measure student knowledge, particularly for 
scores at the margins (e.g., scores at the very low and 
very high ends of the score distribution), a correction for 
measurement error in past test scores is made to ensure 
there is no attenuation bias on the coefficients on past 
achievement. After the regression is run, the school or 
classroom effects are centered to have a mean of zero. 
They are also “shrunk” to ensure that small schools or 
classrooms are not overrepresented among those with 
very high or low value-added. 

VARC currently produces value-added results at the 
district, school, and grade level for many of our partner 
districts (including MRS), and at the classroom level for 
selected partners such as the New York City Department 
of Educafion. District, school, and grade-level results are 
depicted in two primary ways, including a “tier” format and 
a two-dimensional “quadrant” schematic. The tier format 
(see Eigure 1, using an MRS report based on WKCE 
value-added results) displays the school (or grade’s) 
value-added effect on scale score gain in relation to a 
reference point (generally the district or state average) in 
a standardized format on a 0-6 scale, with the district (or 
state) average (denoted by a red vertical line) set at 
Tier growth above 3, accordingly, signifies value-added 
growth above the district/state average, while tiers 
below 3 denote below-average growth. The value-added 
productivity estimate for the school/grade appears 
as a square situated within “error bars” that depict the 
confidence interval associated with the estimate; error 
bars are shorter (smaller) with larger sample sizes and 
less variance in gain scores among the students included 
in the year-to-year growth calculations. 



34 The district/state average in VARC reporting is typically centered at 3, rather than 0, to avoid the impression of “negative” value-added (e.g., a school or grade exerted a negative effect on 
student growth, or a decline in scale scores from one year to the next), which would be an unlikely result for a summative (annual) test with a vertical scale such as the WKCE. 
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APPENDIX A: DEVELOPING NEW HIGH SCHOOL 
VALUE-ADDED INDICATORS IN MPS 



FIGURE 1: SAMPLE REPORT FOR MPS TIER VALUE-ADDED 
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The second main way in which value-added results are 
displayed is in a two-dimensional quadrant format, with 
pre-test attainment level (generally expressed in terms of 
the school/grade's percent proficient/advanced) on the 
vertical axis and value-added tier (as described above) on 
the horizontal axis. By combining attainment (expressed 
in relation to the district or state average) and value- 
added data, each grade, school, or district can be placed 
into one of four “quadrants:” 

• Quadrant 1 : high value-added/high attainment 

• Quadrant 2; high value-added/low attainment 

• Quadrant 3; low value-added/high attainment 

• Quadrant 4; low value-added/low attainment 

A sample two-dimensional/quadrant report for an MPS 
school, again based on WKCE results, appears in Figure 
2 . 



Applying the “base” VARC value-added model to examine 
within-year (e.g., September-May) growth on MPS 9th 
grade benchmark assessments produces results which 
can be displayed in the same format as those shown 
in Figures 1 and 2. As of this writing, data for all MPS 
schools which enroll 9fh graders have been prepared, 
and we are preparing to share these data with MPS 
administrative staff to determine next steps. Several 
issues have been identified for discussion, including the 
fact that many MPS high schools enroll relatively small 
numbers of 9th graders and participation rates for the 
benchmark assessment system tend to be lower for 
most schools than is the case for the WKCE. It is also 
worth noting that while the experimental analyses thus 
far have involved within-year (September-May) growth, 
MPS is also interested in examining growth between 
other testing intervals as well (e.g., September-January, 
January-May, May-May, etc.). 
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FIGURE 2: SAMPLE REPORT FOR MPS TWO-DIMENSION/QUADRANT VALUE-ADDED 
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A final note on the experimental grade 9 value-added 
benchmark work in MPS is that VARC’s collaboration 
with MPS in this area (e.g., expanding value-added to 
benchmark assessments) has generated a great deal 
of interest among other Wisconsin school districts 
in simulating these types of analyses on their own 
benchmark data. The most prominent example here to 
date is a demonstration project recently initiated with the 
Racine Unified School District to produce value-added 
productivity estimates based on data from the Measures of 
Academic Progress (MAP), a well-known online adaptive 
test^^ currently administered in nearly half of Wisconsin’s 
425 school districts. Racine has used the MAP to assess 
student growth in reading and math in grades 2-9 since 
2005, and has desired to have MAP data used for value- 
added purposes in much the same way that value-added 
models are being built for MPS using its own benchmark 
system. In addition to its widespread use in Wisconsin 



and across the Midwest, however, the MAP value-added 
pilot project is attractive to VARC and our partner districts 
in that this test has a very large item pool, extensive 
research in vertical scale development, and smaller 
standard errors (due to the adaptive nature of the test) 
than is generally the case for non-adaptive tests such as 
the WKCE. There is substantial interest statewide in the 
findings from and uses of the MAP value-added system 
being developed for Racine, and as such the vision for 
the Racine's work is to build a statewide MAP value- 
added system for other interested districts to join. We 
envision that this would operate in much the same way 
that the Milwaukee Public Schools (MPS) and Madison 
Metropolitan School District (MMSD) served as pilot sites 
for VARC's statewide value-added system built using 
WKCE data, which was completed in June 2009 and has 
been expanded to more than 30 additional districts as of 
this writing in spring 2010. 



35 Adaptive tests are those that adjust the difficulty of items taken by students in order to more precisely estimate what students know. This format differs notably from most NCLB-type 

accountability tests, in which most of the items taken by students are confined to the specific grade level in which the student is enrolled, regardless of whether his/her ability level is above or 
below that grade. 
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DEVELOPING NEW HIGH SCHOOL VALUE-ADDED 
INDICATORS IN MPS 



It is also exciting to report that M PS itself has very recently 
approved a plan to replace its existing benchmark test 
with the MAP beginning with the 2009-10 school year. 

The second high school value-added initiative supported 
by the Senior Urban Education Research Fellowship 
involves an attendance value-added metric tor first- 
time MPS 9th graders. In much the same way that the 
benchmark assessment value-added system described 
above represents an exciting expansion beyond our 
foundational work using annual NCLB-type tests, 
moving value-added into non-cognitive outcomes such 
as attendance is another key step in developing the 
next generation of value-added models. This is because 
in addition to the obvious focus on measuring gains in 
student achievement through a value-added lens, a 
number of non-cognitive measures, including attendance, 
have been a chronic problem tor MPS and other urban 
districts tor some time. As one example, the year-to-date 
attendance rate tor MPS 9th graders in mid-May 2010 
was 77.4%, which is up slightly trom the 76.2% year-end 
tigure for 2008-09. This represents the equivalent of 
more than one full day each week missed by the average 
MPS 9th grader. 

What is not known from looking at district-level data, 
however, is whether individual high schools show 
better results (which is almost certainly true), nor 
whether schools producing better results are doing so 
with similar students In terms of demographics and 
prior attendance histories. To address these questions, 
the value-added attendance initiative compares schools 
based upon the attendance rates of their first-time 9th 
graders using students’ grade 8 attendance as a control. 
The emphasis on first-time 9th graders, as opposed to all 
9th graders, is important due to the high rates of grade 9 
retention in MPS (25-30% annually) noted previously. In 
other words, the model estimates high school productivity 
on the attendance metric using “prior achievement" (in 
the form of grade 8 attendance) as a control, in order 
that schools can be fairly compared as to how they are 
impacting attendance rates for students who enter high 
school with similar prior attendance histories. 



The general attendance value-added model tor tirst-time 
9th graders can be expressed as 



/V 




+ OCg^S^ 



+ £ 



1 



Based upon the following: 

' = grade i attendance rate 

2 

' = predicted attendance from IV equation. 

iS" 

' = school attended in grade i 

^ =etfect ot prior attendance 
OC 

ii = effect of school attended in grade j on attendance 
in grade i 

£ =prediction error 

The current version of the model uses a transformation 
of the attendance variable to better model student 
attendance. The attendance percentage is transformed 
to its corresponding quantile of the standard normal 
distribution. Also, multivariate shrinkage is used on the 
school effects. The shrunk school effects represented 
by ^99 in the above model is the reported 9th grade 
attendance value-added. 

A sample version ot actual results tor the grade 9 
attendance value-added demonstration appears in Figure 
3. For each school, first-time 9th grader students’ “prior 
achievement," in the form of their grade 8 attendance 
histories, is shown on the vertical axis, and the attendance 
value-added etfect estimate for first-time 9th graders is 
shown on the horizontal axis in relation to the district 
average, which is set at zero. For “Sample” High School 
(which is kept anonymous for now because data from this 
simulation have not yet been shared with individual MPS 
high schools), current (2009-10) first-time 9th graders 
had an average 8th grade attendance rate of 87.6%, and 
the attendance value-added tor this school was -5.4. In 
other words, current first-time 9th graders at this school 
have attendance rates that are 5.4 percentage points 
lower than would be expected it these same tirst-time 
9th graders attended an average MPS high school. 
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FIGURE 3: SAMPLE VALUE-ADDED REPORT FOR FIRST-TIME GRADE 9 ATTENDANCE 
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One of the ways in which this analysis and reporting format 
is particularly useful is that it allows for comparisons 
between high schools that (a) have current first-time 
9th graders with similar "prior achievement" (attendance 
histories as 8th graders); and (b) are of similar size. In 
the “live,” interactive version of the report, each school is 
identifiable with a "mouse-over” feature in the reporting 
tool, and the relative size of each school is highlighted 
by producing different-sized bubbles within the graph 
based on student enrollment counts. In Figure 3, for 
example, it is possible to compare Sample High School 
to Comparison High School, since these two schools are 
of similar size and have current first-time 9th graders 
who are similar based on prior (grade 8) attendance. 
In making this comparison, we see that Comparison 
High School Reading generates a positive value-added 
effect for attendance among first-time 9th graders of 
approximately 4 percentage points, compared to Sample 
High School's negative effect of 5.4 percentage points. 



This result, again based on actual data from fhe simulation, 
represents a “spread” in terms of attendance of more 
than 9 percentage points between two high schools that 
are observably similar along these two key dimensions 
- a substantial discrepancy that certainly merits further 
examination. 

The value-added attendance model for first-time MRS 
9th graders was demonstrated to MRS administrative 
staff in January 2010, and was mef with great interest. 
Among other potential uses for this type of data, the 
district indicated that it would investigate incorporating 
the attendance value-added metric into future decisions 
regarding the approval and renewal of charter schools 
at the secondary level, as well as potential closings and 
consolidations of high school sifes in future years. There is 
also interest in incorporating the first-time grade 9 value- 
added attendance metric into future versions of a school 
ratings system being developed for all MRS schools as of 
this writing in late spring/early summer 2010. 
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